Skip to content
Registry StackDocsv0.38.0

Retain, erase, and settle

For the operator

View as Markdown

You operate a running Registry Casework deployment, hold its migration database credential, and need to account for what the deployment keeps, remove the local payload copies a source owner has authorized you to remove, or close out a source attempt whose outcome Casework could not observe. Casework expires unified review state on its policy clocks and keeps source-backed work until you act. The operator commands here each change one exact selection, take no copy first, and write nothing until you repeat them with --apply.

Two retention regimes run side by side. Unified reviews carry declared periods and the runtime enforces them. Source-backed work carries no clock at all: it stays until an operator erases it for one exact source request.

StateKept forWhat removes it
Review request context, result and constraints, notes, drafts, tasks, and minimized historyterminalDays from terminal settlementThe runtime retention pass
Review clock occurrences, including a subject clock a changes-requested settlement pausedThe same terminalDays from terminal settlement of the round the clock is bound to; a later round of the same review kind inside that window continues the subject clock and takes over the binding, and a round after it starts a fresh subject clock with a full deadlineThe runtime retention pass
Producer result-feed eventsThe same terminalDays from terminal settlementThe runtime retention pass
Protected accountability record: the raw issuer-qualified deciding identity, the staff reason, and only a sha256 digest of a recorded resultaccountabilityDays from the same terminal time, never shorter than terminalDaysThe runtime retention pass
The request’s producer and initiator identitiesRaw until terminalDays, then a request-bound sha256 tombstone (not keyed with a secret) until the accountability deadline, so the same producer or initiator still learns the result expiredThe runtime retention pass
Review-create and mutation replay responsesUntil request payload expiry, then a payload-free tombstone until the accountability deadlineThe runtime retention pass, after which the key is free again
Source-backed subjects, work items, private drafts, attempts, receipts, history, and durable eventsNo expiry; they stay for the life of the databasecaseworkctl retention erase --apply
The audit file and its sealed filesSealed files for audit.retainDays (90 by default); shipped copies for your retention policyThe runtime’s audit writer; your own tooling for shipped copies

Each review kind declares terminalDays and accountabilityDays in the project, and the project is refused unless terminalDays is at least 1, accountabilityDays is at least terminalDays, and accountabilityDays is at most 3,650. Pin the two periods to the institution’s own decision record: Casework enforces the numbers it was given and holds no view on what they must be.

The retention pass runs inside casework serve, on every thirtieth tick of the two-second maintenance loop, so about once a minute. It works in bounded batches of at most 100 rows per table per pass, skipping locked rows, which keeps it from competing with live traffic and means a large backlog drains over several passes rather than in one. It deletes expired result-feed events, request context, results, notes, drafts, tasks, history, clock occurrences, and replay responses, then deletes the protected accountability record and payload-free replay tombstones at their later expiry. A failed pass logs a warning and the next tick retries.

A structured decision result is decision data, not source record content, but it follows the same clock as the request context it was decided on: the result is deleted at terminalDays, together with the constraints the producer supplied at create time. What outlives them is the accountability record’s sha256 digest of the result, which proves that a specific result was decided without keeping its content until accountabilityDays closes.

Producer clients must poll inside the terminal period, because an expired request is no longer reachable through the request read, result, history, notes, tasks, or result feed. When a review request payload expires, Casework erases the stored idempotency response and keeps only a request hash and hashed binding metadata in a tombstone. An exact retry against that tombstone returns the idempotency.expired problem with HTTP 410, so the caller reconciles the original operation instead of creating a second request; a changed request under the same key still returns idempotency.key-reused. Once the accountability deadline passes, the tombstone is forgotten and the key may be reused.

Run these commands on the operator host, where the secret root and the migration database credential are readable. Each one loads runtime.yaml in the project directory unless you pass --runtime-config. That file’s package.root selects the authored casework.yaml; inspect it before applying a command. Each connects with the migration credential, not the runtime credential.

Each command previews by default and writes nothing until you repeat the exact invocation with --apply. These examples use --format json so you can compare the exact selector, counts, and applied state in the preview and apply reports. A successful run exits 0. A refusal puts {"ok": false} with a diagnostic on standard output and exits 1 for an invalid project or configuration, or 3 for an operational failure. Without --format json, Casework prints a human-readable result and sends refusal diagnostics to standard error.

An --apply run writes audit entries the way the service does, to the caseworkctl companion of the configured destination: with audit.path set to .../casework.ndjson, that is .../casework.caseworkctl.ndjson in the same directory, on the host where you run the command, with its own lock, rotation, and retention. The service keeps its own file while the command runs. Run the command as a user the audit directory accepts, and ship the companion file with the service’s. When the companion file cannot be opened, the apply run refuses before it changes anything. A stdout destination writes the entries to the command’s standard error instead, so the command’s own report keeps standard output; collect standard error to ship them.

A source owner decides what an authoritative record keeps. This command removes only the copies Casework holds locally about one exact source request, named by its source identifier, request kind, and request identifier. Preview it first:

Terminal window
caseworkctl --format json retention erase /etc/registry-casework/package \
--runtime-config /etc/registry-casework/runtime.yaml \
--source-id professional-register \
--request-kind "<request-kind>" \
--request-id "<request-id>"

The report is counts and the selector you passed, with no work item identifiers and no erased values: items, drafts, correctionContexts, attemptPayloads, receiptPayloads, historyDetails, eventDetails, idempotencyResponses, clockOccurrences, and clockPreviews, beside blockedLiveAttempts and applied. Read blockedLiveAttempts before anything else: a nonzero count means a pending or uncertain source attempt is still open on this request, and the apply run will refuse rather than erase, so that Casework never buries an action whose outcome nobody has established. Recover or settle that attempt first. Then check the counts against the source owner’s retention decision.

Terminal window
caseworkctl --format json retention erase /etc/registry-casework/package \
--runtime-config /etc/registry-casework/runtime.yaml \
--source-id professional-register \
--request-kind "<request-kind>" \
--request-id "<request-id>" \
--apply

One transaction marks the local subject inactive and the items erased, deletes the private drafts and the correction context, strips the decision reason, flagged fields, displayed binding, recovery evidence, and receipt metadata from settled attempts, empties the detail of every history row and durable event row, drops the stored idempotency responses, and cancels the outstanding clock work for that request. It keeps the rows themselves as bounded tombstones, so the item lifecycle stays accountable while its values are gone. The apply run writes a casework.source_retention_erased request entry under the profile system:operator before it opens the transaction, and a response entry naming the same event after it commits; neither entry carries the selector or the counts. The apply report repeats the preview counts with applied set to true.

Erasure stays local to this deployment: the command reaches no Base Registry Engine (BReg) instance, sends no request to the source, and does not erase entries already written to the audit destination. Those entries name items only by keyed pseudonyms and carry no payload values.

A completed apply run exits 0 and reports applied: true. Re-running the preview afterwards is the check: the payload counts come back as zeros, because nothing selectable is left to erase.

A refusal names its cause without naming the request. a source attempt is still pending recovery means an attempt on this request is pending or uncertain; recover it through the holder’s client, or settle it, then run the erasure again. the requested resource was not found means no local subject matches the three selector values, which is the usual shape of a typo in one of them. the request is invalid means one selector value is empty, longer than 512 bytes, or carries a control character. If the selected package is unexpected, inspect package.root in the selected runtime configuration and pass the intended file.

An attempt turns uncertain when Casework sent a decision to the source and could not confirm what happened to it. The holder’s client recovers such an attempt by replaying it, and periodic readback of the source repairs most of the rest. What remains is the case where neither can observe the outcome. Establish with the source owner what the source actually did, then preview the settlement:

Terminal window
caseworkctl --format json attempt settle /etc/registry-casework/package \
--runtime-config /etc/registry-casework/runtime.yaml \
--attempt-id "<attempt-id>" \
--outcome not-applied \
--reason "<what the source owner confirmed, and how>" \
--decided-by "<who decided>"

The attempt identifier is the one Casework returned in the Registry-Casework-Attempt header beside the work-item.recovery-pending problem the holder saw. Use --outcome applied when the source applied the decision and --outcome not-applied when it did not. The preview takes the same locks and runs the same checks as the settlement, then rolls back: it reports attemptId, itemId, operation, bindingReference, your outcome, reason, and decidedBy, and the attemptState and itemState the settlement would produce, with applied still false. The bindingReference is a sha256: digest over the subject and the displayed binding, not a source value.

--reason accepts at most 2,000 bytes and --decided-by at most 256 bytes; both must be non-empty text without control characters. Both are recorded verbatim in the work item’s history, which every Staff member and Supervisor serving that queue can read, so write them for that audience.

Terminal window
caseworkctl --format json attempt settle /etc/registry-casework/package \
--runtime-config /etc/registry-casework/runtime.yaml \
--attempt-id "<attempt-id>" \
--outcome not-applied \
--reason "<what the source owner confirmed, and how>" \
--decided-by "<who decided>" \
--apply

One transaction changes the attempt, its work item, and the item’s history together. not-applied refuses the attempt and returns the work item to its holder, who can act on it again. applied completes the attempt with no source receipt and leaves the work item synchronizing, and marks the subject for synchronization so the next source observation reconciles it. Either way the run writes a fresh execution token, which fences any executor still holding the old one, and appends one attempt_settled history event carrying the attempt, binding reference, operation, outcome, reason, and decider, with no actor, because a settlement is an operator act rather than a caseworker’s decision. This command reaches no BReg instance.

A completed apply run exits 0 and reports applied: true with the states the attempt and the item now hold. Confirm the same states through the work item’s history, where the attempt_settled entry now sits.

Four refusals cover the cases worth planning for. the source attempt still holds a live execution lease; wait for it to expire means a client acquired the attempt within the last 330 seconds and may still be completing it; wait, then preview again. the source attempt is completed (or refused, or pending) means only an uncertain attempt can be settled, and a terminal attempt already has its outcome. Recover a pending attempt first: once its execution lease expires, the actor who started it calls POST /v1/work-items/{itemId}/attempts/{attemptId}/recover while the source is reachable. Recovery completes or refuses the attempt from the source’s answer, and leaves it uncertain only when the source cannot give one; settle it only then. When that actor cannot recover it, mark it uncertain first. no source attempt has this identifier means the UUID names nothing in this database. the work item is claimed (or another state) means the item is no longer awaiting its source outcome, so recovery or readback has already resolved it and no settlement is needed.

Only the actor who started a pending attempt can recover it: the recover route is bound to that actor’s issuer, subject, and Casework profile, and no Supervisor or Administrator can recover it in their place. When that actor cannot, because they left or their client is gone, the attempt stays pending and its work item stays synchronizing. Once the attempt’s 330-second execution lease has expired, mark it uncertain and then settle it as above:

Terminal window
caseworkctl --format json attempt mark-uncertain /etc/registry-casework/package \
--runtime-config /etc/registry-casework/runtime.yaml \
--attempt-id "<attempt-id>" \
--reason "<why the actor who started it cannot recover it>" \
--decided-by "<who decided>"

The preview takes the same locks and runs the same checks as the marking, then rolls back. It reports attemptId, itemId, operation, bindingReference, the originalActor issuer and subject with their originalProfileId, your reason and decidedBy, and the attemptState and itemState the marking would produce, with applied still false. --reason and --decided-by carry the same limits as a settlement and land in the same work item history, so write them for the Staff members and Supervisors serving that queue.

Repeat the run with --apply. One transaction moves the attempt to uncertain, writes a fresh execution token that fences the original executor out, leaves the work item synchronizing, and appends one attempt_uncertain history event with no actor. Its detail carries the attempt, binding reference, and operation, your text as operatorReason and decidedBy, and the originalActor and originalProfileId, so the history names both the operator’s decider and the person whose attempt it was. The marking decides nothing about the source outcome and reaches no BReg instance: settle the attempt once the source owner confirms what the source did.

the source attempt still holds a live execution lease means the lease has not expired yet; wait, then preview again. the source attempt is uncertain; only a pending attempt can be marked uncertain means the marking is not needed; settle the attempt. completed or refused in its place means the attempt already has its outcome. the work item is claimed (or another state) means the item no longer awaits its source outcome, so nothing is stranded.

Casework writes each audited operation a caller requests through the shared audit writer: a request entry before the operation’s transaction opens and, after it commits, one response entry per recorded event, or a single response entry recording a replay or a no-op. They share one correlation and carry the schema registry-casework-audit/v1. Background work that no caller requested, such as a reconciliation pass, writes only response entries and starts only while the writer is ready. Entries name work items, grants, teams, queues, and principals only by keyed pseudonyms, and carry no payload values. They are not chained or signed.

With the default file destination, one process holds a lock beside audit.path and appends to it. When the next entry would take the file past audit.rotateBytes (100 MiB by default), the writer seals it under an ascending eight-digit suffix (casework.ndjson.00000001, casework.ndjson.00000002, and so on) and opens a fresh file at the configured path. At startup and on each rotation it deletes sealed files older than audit.retainDays (90 by default, at most 36500). Nothing on the host makes the file tamper-evident and nothing keeps it past retainDays, so ship sealed files to append-only storage you operate before they age out, and apply your retention policy to the shipped copies. With destination: stdout, each entry is one line on standard output, and collection and retention belong to the platform that reads the stream.

The pseudonyms are keyed with the audit secret named by the runtime file’s audit.hashKeyRef, so a reader of the audit file who lacks the secret cannot turn a pseudonym back into the identifier it replaces by hashing candidates. Provide at least 32 bytes of NUL-free text through a secret provider, never in the runtime file:

Terminal window
(umask 077 && openssl rand -hex 32 > /run/secrets/registry-casework/casework-audit-key)
chmod 0400 /run/secrets/registry-casework/casework-audit-key

A file secret sits immediately below the configured secret root, belongs to the runtime user, and carries mode 0400 or 0600 with no symlink and no second hard link. Its exact bytes are the key, including a trailing newline, so rewriting the file with a different byte changes every pseudonym written after it. An empty value, a value over the provider’s size limit, or one carrying a NUL byte is refused without echoing it. caseworkctl doctor resolves the reference as a readiness check and reports success without printing the bytes.

A write the destination refuses fails the audited call with service.unavailable and stops the writer for the rest of the process’s life, so GET /ready answers 503 and every later audited call is refused while the process keeps running. The runtime log carries audit file write failed with the cause. Treat it as a disk, permission, or lock problem at the audit path rather than a database outage, fix it, and restart the process.

SymptomNext move
The erasure preview reports a nonzero blockedLiveAttemptsA pending or uncertain attempt is open on that request. Recover it through the holder’s client, or settle it, then preview again.
retention erase --apply refuses with a source attempt is still pending recoveryAn attempt reached one of those states between your preview and your apply run. Resolve it and repeat both runs.
retention erase refuses with the requested resource was not foundNo local subject matches all three selector values. Confirm the source identifier, request kind, and request identifier with the source owner.
A command inspects an unexpected Casework packageCheck package.root in the selected runtime configuration and pass the intended --runtime-config file.
attempt settle refuses with the source attempt still holds a live execution leaseA client acquired the attempt within the last 330 seconds. Wait for the lease to expire, then preview again.
attempt settle refuses with only an uncertain attempt can be settledThe attempt is pending, completed, or refused. A terminal attempt already carries its outcome. Recover a pending one first: once its 330-second execution lease expires, the actor who started it calls POST /v1/work-items/{itemId}/attempts/{attemptId}/recover while the source is reachable. Settle only if recovery leaves it uncertain. If that actor cannot recover it, mark it uncertain with attempt mark-uncertain, then settle it.
attempt mark-uncertain refuses with only a pending attempt can be marked uncertainAn uncertain attempt needs no marking; settle it. A completed or refused attempt already carries its outcome.
attempt settle refuses with only a work item awaiting its source outcome can be settledRecovery or source readback already moved the item out of synchronizing. No settlement is needed.
attempt mark-uncertain refuses with only a work item awaiting its source outcome can have its attempt marked uncertainRecovery or source readback already moved the item out of synchronizing. No marking is needed.
A review request disappeared from producer polling and the result feedIts terminalDays period closed. The accountability record may still exist; an authorized Supervisor can resolve one event until accountabilityDays closes too.
An exact retry returns idempotency.expired with HTTP 410The item payload expired and only a payload-free tombstone remains. Reconcile the original operation before choosing a new key.
GET /ready fails with audit file write failed in the logThe audit path, its free space, its permissions, or its single-writer lock is the likely cause. Fix it, then restart the process.
retention erase --apply or attempt settle --apply stops at opening the Casework operator audit destinationThe caseworkctl companion file beside audit.path cannot be opened on this host, or another caseworkctl run holds its lock. Run the command as the runtime user on a host where the audit directory exists, and one at a time.