Retention and persistent state

Use this page to decide which Registry Stack state needs backup, off-host shipping, expiry, or deletion policy outside the products.

Operator boundary

Registry Stack gives operators rotation knobs, TTL-bound protocol stores, and pseudonymized audit handles. It does not decide the legal retention period for a deployment, delete source-registry records, or prove that a local rotating file kept every historical audit event.

Two consequences follow from that boundary:

• Treat local audit files as neither tamper-evident nor complete: they are not chained, and retention deletes sealed files by age. Ship them to append-only storage when integrity or completeness matters.
• Treat keyed pseudonyms as a linkability control, not as erasure. Destroying or rotating the HMAC secret prevents recomputing old pseudonyms from raw values, but it does not delete audit records, cache entries, or source data.

Durable state and externally retained records

Store	What it contains	Operator control
Evidence Gateway audit file	Value-free access, release, denial, and failure events	Configure rotation and retention, and ship sealed files before local expiry.
Registry Render audit file	Value-free render events and artifact identities	Apply the same shared-writer handling as Evidence Gateway.
Base Registry Engine PostgreSQL database	Current records, retained revisions, requests, access logs, and webhook delivery state	Back up the database and use the owned erasure and retention commands.
Operator-owned files	Runtime configuration, packages, sources, checkpoints, exports, and secret references	Protect, back up, and expire these files through platform controls.

Process-local caches and client-held state

Evidence Gateway can keep OIDC signing keys and negative lookup entries in process. Restarting the service clears that cache. It stores no source records or assertions in the cache. Client-held continuity receipts and verification material remain the relying application’s responsibility.

Audit retention

Evidence Gateway, Registry Render, Base Registry Engine, Casework, and Scheduling write through the shared per-process audit writer. The active file rotates at the configured size and sealed files expire by age. A stdout destination delegates both controls to the log collector.

The age is set in whole days, from 1 to 36,500, and is 90 when omitted. Each of these runtimes reads it from audit.retentionDays in its runtime.yaml. The rest of this page calls the value the audit retention period.

Never truncate, rewrite, or reserialize the active audit file as part of cleanup: the writer treats any external change to it as a fault and stops, and audited requests fail closed until a restart. Copy sealed files to append-only storage before the audit retention period deletes them.

A crash, a killed process, or a full disk can leave the active file’s final line incomplete (any byte other than a newline at end of file). The next start recovers it without operator action: the writer copies the bytes after the last complete line to the owner-only (0600) side file <audit.path>.torn, syncs it, truncates the active file to its last complete line, and logs the side file’s path and byte count, never its bytes, at error level. An entry is acknowledged only after the write holding it is synced, so the torn bytes belong to no acknowledged entry and no acknowledged entry is lost. doctor and companion checks accept a destination the writer would recover this way and change nothing.

Archive the side file with the sealed files, then remove it. The writer never overwrites it: if a later torn line finds <audit.path>.torn holding other bytes, the start is refused until that file is moved. A tail longer than any entry, or a file whose first entry is in another format, is still refused unchanged; archive that file and restart with a fresh audit.path. Do not edit the active file to repair it; editing it while a writer holds it is the same external-change fault described above.

Evidence Gateway retention

Evidence Gateway has no application database and persists no selector, source, evidence, or response data. Its durable state is its governed signer configuration and public keys plus the audit file described in Durable state and externally retained records. The external Transit service retains the production private key. An external durable audit service may own its own storage, but Evidence Gateway itself does not maintain one.

The runtime recognizes a sealed file by its <audit.path>.<sequence> name, with an eight-digit sequence, and never deletes the active file. Retention deletes sealed files in sequence order, oldest first, and stops at the first one whose last modification is inside the audit retention period, so the local directory holds a rolling window, not the whole history. A deletion that fails is logged and retried at the next rotation or start; it never stops the writer. Each stream reserves every <audit.path>.<sequence> name, plus <audit.path>.lock, <audit.path>.seq, <audit.path>.seq.tmp, and <audit.path>.torn, and a path whose file name ends in one of those suffixes is refused at startup, so no stream can be pointed at another’s file. The .seq file records the next sequence at every start and before every rotation, so a restart after retention or a shipper removed every sealed file continues the numbering instead of reusing a name. Read and ship the Evidence Gateway audit log describes the entry format and the stdout destination.

Ship Evidence Gateway audit records off host

The local audit file is not tamper-evident, and retention deletes it by age, so the append-only store you ship to is the audit record of the deployment. A shipper reads the same directory the runtime writes, so three rules keep it safe:

• Copy sealed files, and ship each one before the audit retention period deletes it. The runtime never depends on a sealed file after sealing it, so removing one after it is archived is safe.
• Key archived copies on more than the file name, for example the host and the first entry’s time. Sequence numbers continue across restarts only while <audit.path>.seq survives, so a fresh or restored directory can seal a file under a name already shipped.
• Never point a shipper at the active file in a way that renames, truncates, or rewrites it, and never touch <audit.path>.lock or <audit.path>.seq. Ship <audit.path>.torn, when one appears, like a sealed file, then remove it. Any external replacement or modification of the active file makes readiness and later appends fail closed. Re-applying its owner-only mode, ACLs, or extended attributes, as configuration management does, is not a modification.

Archive the oldest sealed files first, for example:

cp /var/lib/registry-evidence/audit/evidence.jsonl.00000001 "$ARCHIVE/<host>-<first-entry-time>.jsonl"

Alert on the shipper, not on the local directory: its size stays bounded by retention even when a shipper has stopped, and local files that retention deletes before shipping are lost.

Render retention

Render keeps no application database, no source data, and no rendered document beyond the response it returns; a render is a pure function over a sealed template bundle and the request body. Its only durable, product-owned state is the sealed bundle on the operator host and the audit file described in Durable state and externally retained records.

The runtime recognizes a sealed file by its <audit.path>.<sequence> name and never deletes the active file. Retention deletes only sealed files whose last modification is older than the audit retention period, so the local directory holds a rolling window, not the whole history. Ship sealed files the way Ship Evidence Gateway audit records off host describes; the mechanics are identical.

A release before 2026-09-25 kept its own keyed hash chain and exposed registry-render audit-verify to check it, and its runtime file accepted directory, integrityKeyRef, and maxSegmentBytes. The shared-writer release refuses all three as unknown fields and has no audit-verify, since the new log carries no chain to verify. Before the first start on the shared-writer release, archive the old chained log to append-only storage and point audit.path at a fresh file in a directory that holds no old segments; do not point the new runtime at the old chained file or directory.

Upgrade Casework and Scheduling audit

v0.40.0 does not upgrade v0.39.0 state in place; apply to a new database. Fresh installation creates the current schema without converting or discarding earlier audit rows. plan and apply retain their unpublished-audit refusals and leave earlier outboxes untouched.

If you retain an earlier PostgreSQL audit-outbox deployment, stop new traffic and keep its runtime running until its publisher has delivered every pending audit row. For an earlier database that has one of these outbox tables, check the pending count with your database tooling:

-- Casework database
SELECT count(*) FROM casework_audit_outbox WHERE published_at IS NULL;
-- Scheduling database
SELECT count(*) FROM scheduling_audit_outbox WHERE published_at IS NULL;

When the applicable count reaches zero, stop the old runtime and archive its audit file, sealed segments, and audit key outside the new writer’s retention directory. Keep the earlier database with the release that wrote it until its work and accountability records are exported. Do not delete pending rows to bypass a refusal.

Configure a fresh audit file path and a new database for the new runtime. Run caseworkctl apply or schedulingctl apply with this release’s runtime configuration pointing at that new database.

The new release writes directly to the shared per-process writer. Its audit entries are not chained. Collect the service files and operator companion files, and use append-only storage for tamper evidence. See earlier database and audit state for the Casework state and audit boundary.

Base Registry Engine retention

A Base Registry Engine registry keeps record history and the webhook outbox in PostgreSQL. Its audit stream is separate: back up the database and ship the per-process audit files, including operator companion files, through your log pipeline. A database restore does not restore or erase those files.

registry_internal.registry_revisions holds the retained revisions of a record and registry_data holds its current row. bregctl history erase is the only command that removes retained revisions. It erases them for one record up to a revision number, scrubs the affected correction context and retained outbox payloads, and replaces the affected cached idempotency responses with refusal tombstones, so a retried mutation key cannot replay an erased body. It leaves the current record, every later revision, and the audit journal in place. Snapshot references at or after the earliest erased commit become unavailable until bregctl history rebaseline writes a new baseline commit.

Change-request detail is separate from record history. bregctl request-retention erase removes the proposal and target snapshots, request revision snapshots, held idempotency responses, outbox payloads, and intake rows of one proposal version, and leaves the value-free lifecycle rows of the request. It applies only where the request entity declared retention.mode: operator-erase; the default, retain, keeps every proposal version and refuses operator erasure.

Spent idempotency keys live in registry_internal.registry_idempotency, one row per key, found by a digest of the verified issuer, principal value, and key. Until its receipt is dropped, the row also holds those raw values. Neither erasure removes that row or its raw values: both drop or tombstone only the held response, so a retry answers 409 idempotency.conflict. A held response stays replayable for idempotency.receiptRetentionDays, 7 days by default, and bregctl idempotency-retention erase-expired drops the ones past that horizon, together with the raw issuer, principal value, and key, while the digest keeps the key spent. The digest row is kept indefinitely, because the results and application of an immediate action hang off it and revision history reads its provenance from them. Drop expired idempotency responses gives the command and its limits.

Audit retention uses the shared file writer described in Audit retention. History erasure leaves those files untouched. The database records committed erasure coverage in a dedicated table, so deleting an old audit segment does not affect successor-package admission.

The webhook outbox also has scheduled payload expiry. The runtime clears a retained payload on the first successful delivery, and clears an undelivered payload once its expiry has passed, which is also the point after which that delivery can no longer be replayed. The delivery rows themselves stay; only the payload goes.

An entity that declares accessLog writes one row per logged record read to registry_internal.registry_subject_access_log. Each row takes its expiry from the entity’s retentionDays (default 90) when the read happens, and the subject’s access-log route hides it once that expiry passes. A background job erases expired rows every minute in batches of 1000 until none remain, up to 100,000 rows a minute; a run that stops at that bound logs a warning with the remaining backlog. The job keeps running after a later package turns accessLog off, and a package change never extends an existing row’s expiry. Database backups keep their own copies of these rows.

Ingestion runs live in registry_internal.registry_ingestion_runs, with their chunks and chunk receipts beside them. A run row holds the verified issuer and principal value of the caller that created it, which decide who may list, read, continue, or cancel the run, and runs have no retention horizon: no command removes a run or its creator values. bregctl history erase erases the chunk receipts that describe the erased revisions and leaves the run and its counts.

Import and export checkpoints live on the operator host, not in the database. bregctl data import writes a checkpoint file beside a .state sidecar it owns, and bregctl data export writes its output file beside a checkpoint that binds the package revision, schema fingerprint, entity, profile, and operation. Both refuse a run whose files no longer match rather than resuming it, so keep each pair together and unedited or start both afresh. An export output holds record values and needs the handling any other extract gets.

Package files are operator-owned and nothing in Base Registry Engine expires them. Keep the project sources, module locks, generated files, test receipt, signature documents, and runtime file for every activation, and store secrets separately.

Spent idempotency keys

Base Registry Engine, Registry Messaging, and Registry Scheduling each keep a row for every idempotency key a caller spent, so a retry is answered rather than run again. Until the product’s receipt horizon, that row holds the raw issuer, subject, and key beside the held response. After each product’s receipt horizon the raw caller and key are cleared and a digest keeps the key spent: a retry under it is still refused, and the same key sent by another caller is that caller’s own fresh key. Each product refuses it by the same rule: the same request is 410 idempotency.expired, and a different request is a 409 idempotency conflict, as it is inside the horizon. Messaging keeps the request hash with the message record, so once it deletes the record after retention.recordRetentionDays any request under the key is 410. Base Registry Engine clears the raw caller and key when the operator runs bregctl idempotency-retention erase-expired; Messaging and Scheduling clear them in their retention sweeps.

What remains outside the product

Registry Stack does not implement a general erasure workflow for source records, audit records, or cache entries, beyond the bounded Base Registry Engine commands described in Base Registry Engine retention. It also does not prune operator-owned config files, source files, or old release artifacts after the product-level expiry checks have run. Those controls belong in the operator’s storage, backup, incident-response, and data-protection processes.

Next

• Retain, erase, and audit
• Harden a production deployment
• Known limitations and non-guarantees