Retention and persistent state

View as Markdown

Use this page to decide which Registry Stack state needs backup, off-host shipping, expiry, or deletion policy outside the products. It documents current Registry Relay, Evidence Gateway, and Registry Mint behavior. Retention enforcement, record-level erasure, and data-subject-rights workflows are outside the 1.0 product boundary.

Operator boundary

Registry Stack gives operators rotation knobs, TTL-bound protocol stores, and pseudonymized audit handles. It does not decide the legal retention period for a deployment, delete source-registry records, or prove that a local rotating file kept every historical audit event.

Two consequences follow from that boundary:

Treat local audit files as tamper-evidence for the retained records, not as a completeness guarantee for all records ever written. Use off-host shipping when completeness matters.
Treat keyed pseudonyms as a linkability control, not as erasure. Destroying or rotating the HMAC secret prevents recomputing old pseudonyms from raw values, but it does not delete audit records, cache entries, or source data.

Durable state and externally retained records

Store	What it can contain	Expiry or rotation	Operator control
Relay audit sink	Chained attempt, terminal, and refusal envelopes binding the registry, resource, operation, access profile, disclosure profile, selected-field identifiers, contract revision, and source revision. It excludes tokens, selectors, raw principals, source values, response values, and raw identifiers.	Relay uses the same segmented, hash-chained JSONL sink Evidence Gateway and Registry Mint use. The active segment seals at a fixed `64 MiB` threshold and reopens under the next ascending sequence; Relay does not expose a configurable segment size or a retained-file count, and it never deletes a sealed segment itself.	Configure `audit.sink` and `audit.integrityKeyRef` in `runtime.yaml`. Archive sealed segments off host oldest sequence first; the shipping mechanics are identical to Ship Evidence Gateway audit records off host below, since both sinks share the same segment-naming and hash-chain scheme.
Evidence Gateway audit chain	A keyed, hash-chained JSONL log of the access-attempt event before every source read and the disclosure-release event before every response. Audit carries reviewed identifiers and decision categories, never raw selector values, source values, credentials, tokens, or raw subject identifiers.	`auditStorage.maximumFileBytes` is a per-segment rotation threshold, not a total ceiling. When an append would exceed it, the runtime seals the active segment as `<auditStorage.path>.<sequence>` and opens a new active segment online, with no operator action. Nothing in the runtime deletes or compacts a sealed segment; total retention is entirely operator-owned.	Set `auditStorage.path`, `auditStorage.maximumFileBytes`, and the hash secret in `runtime.yaml`. Archive sealed segments to cold storage oldest sequence first, never touch the active segment while the service runs, and run `evidence verify-audit` on the cadence the deployment’s audit policy requires.
Mint audit chain	A keyed, hash-chained JSONL log of token releases and denials. Records use keyed pseudonyms and omit raw assertions, access tokens, client ids, principals, authority values, actors, and subject values.	`audit.maximumFileBytes` is a per-segment threshold, not a total ceiling. Registry Mint seals full segments as `<audit.path>.<sequence>` online and never deletes or compacts them.	Set `audit.path`, `audit.maximumFileBytes`, and the hash key in `mint.yaml`. Archive sealed segments oldest first, never touch the active segment while Registry Mint runs, and run `mint verify-audit` on the deployment’s verification cadence.
Operator-owned config, source, and secret paths	Relay’s `runtime.yaml`, the sealed package at `packagePath`, and the bound SQLite source path; Evidence Gateway’s governed bundle and runtime document; Registry Mint’s governed public keys, signer configuration, and client registry; secret references for all three. Source files can contain personal data.	Registry Stack does not expire these files, except through the specific audit and cache mechanics above.	Mount source data read-only where possible; back up config, packages, and secrets through your platform controls.

Relay V2 keeps no other durable, product-owned state beyond the audit sink and the operator-owned paths above: no shipper-acknowledgement cursor file, no server-side ingest cache, no configuration-signing or anti-rollback state, and no separate consultation database. Relay V1 had all four; the Relay recovery state section explains why none of them carry forward.

Process-local caches and client-held state

Store	What it can contain	Expiry or rotation	Operator control
Relay OIDC JWKS cache	Issuer signing keys and negative lookup entries, not subject records. Applies only when `authentication.issuer` is configured; a package where every access profile is public can run with `authentication.issuer: null` and no cache at all.	In process only. Relay V2 exposes no cache-lifetime override, so the shared platform defaults apply: `600` second positive cache TTL and `60` second negative cache TTL.	Restart clears the cache.
Evidence Gateway OIDC JWKS cache	Issuer signing keys and negative lookup entries for the configured `authentication.jwksUri`, not subject records.	In process only. Evidence Gateway exposes no product-level override, so the shared platform defaults apply: `600` second positive cache TTL and `60` second negative cache TTL.	Restart clears the cache. An unreachable key set is retried and reported at a bounded interval rather than silently ignored.
Relay response headers	No stored response cache. Relay validates certain public snapshot responses with a strong `ETag` and answers a matching conditional request with `304 Not Modified`; every other response, including metadata, defaults to `Cache-Control: no-store`.	Not time-bound; validators change when the underlying package, source revision, or requested representation changes.	Not configurable. Behavior follows the source profile and access profile compiled into the package.
Relay pagination cursors	Client-held, authenticated, and encrypted cursor payloads binding the source and contract revisions, operation, access profile, disclosure profile, filters, order, selected fields, and authorization context. No plaintext filter, order, or bbox value.	Not stored server-side. Each cursor carries an expiry checked against `cursor.maximumAgeSeconds`, which the runtime defaults to `300` seconds only when the whole `cursor` section is absent, and becomes invalid immediately if the bound request context no longer matches. Cursors apply to snapshot-source list and search operations alike; live read-only sources do not support pagination.	Configure `cursor.integrityKeyRef` and, optionally, `cursor.maximumAgeSeconds` in `runtime.yaml`. Treat cursor tokens as opaque, client-held request context.
Mint client-assertion replay cache	Seen `jti` values from presented client assertions, used to refuse a repeated assertion.	In memory only, bounded by `clientAssertion.replayCacheEntries` (default `8192`, minimum `256`). Restart clears it.	Size `replayCacheEntries` to the deployment’s client population and assertion volume; there is no separate time-based expiry knob.

Relay V2 has no auth-failure throttle and no source-side OAuth token cache: caller authentication is a stateless bearer-token check against the configured OIDC issuer, and a source is a read-only SQLite file with no credential of its own to cache a token for.

Audit retention

Relay writes chained JSONL audit records through the same segmented sink Evidence Gateway and Registry Mint use, sealing full segments at a fixed 64 MiB and never deleting a sealed segment itself.

The chain detects edits, insertions, reordering, and deletion after the first retained record within the retained set. It does not prove that older rotated-away records, or earlier retained records, still exist. Off-host durability is entirely the operator’s responsibility: Relay V2 has no deployment-posture report, no evidence_grade startup mode, and no doctor command, so it never surfaces a finding about local-only retention risk the way Relay V1 did. There is also no shipper-acknowledgement cursor to check a shipper’s progress against, and no relay verify-audit or relayctl command that replays the chain: unlike Evidence Gateway’s evidence verify-audit and Registry Mint’s mint verify-audit, Relay V2 has no offline chain-integrity command. Verify continuity by inspecting the sealed segment sequence directly, and treat a running deployment’s /ready result as evidence the sink was writable at that moment, not as proof of complete off-host retention.

Never truncate, rewrite, or reserialize audit files as part of cleanup. Back them up as files, keep old integrity keys for the audit retention period, and rotate audit.integrityKeyRef only as an intentional key-lifecycle event.

Evidence Gateway and Registry Mint retention

Evidence Gateway has no application database and persists no selector, source, evidence, or response data. Its durable state is its governed signer configuration and public keys plus the audit chain described in Durable state and externally retained records. The external Transit service retains the production private key. An external durable audit service may own its own storage, but Evidence Gateway itself does not maintain one.

Run evidence verify-audit against the runtime document’s configured audit path to check chain integrity across every segment. It replays every segment across every seam and reports segments, records, sealed-sequence, head, and active-segment; a gap in the sealed sequence is reported as a distinct missing-segment result rather than treated as silent history loss. Archive sealed segments to cold storage oldest sequence first and never rename or otherwise touch the active segment while the service runs: the runtime recognizes a segment by its <auditStorage.path>.<sequence> name, and a rename outside that namespace produces a silent fork on the next restart rather than a rotation.

Ship Evidence Gateway audit records off host

An external receiver is a mirror of the local chain, not a replacement for it. Evidence Gateway recovers its chain head from local segments at startup and holds that head in process memory, so a write-only remote sink cannot supply the previous hash a restart needs. Ship copies and keep the local audit path as the system of record.

A shipper reads the same directory the runtime writes, so two rules keep it from corrupting the chain it is copying.

Copy sealed segments, never move them, and leave the newest sealed segment in place. When the runtime opens the audit path it derives the next sequence number from the newest sealed segment still on disk. A restart after a shipper deleted that segment normally fails closed, because the active segment’s first record names a predecessor the remaining sealed tail no longer matches. Nothing catches it when the active segment is empty at that restart, the state every rotation leaves behind until the next record arrives: startup accepts the older sealed tail, and a later rotation seals a fresh segment under the sequence number the shipped-away segment already used. Local verification stays clean, since the sealed numbering is still dense, and cold storage ends up holding two files named <auditStorage.path>.00000001 that hold different records and claim the same predecessor.
Never point a shipper at the active segment or at <auditStorage.path>.lock. Any external replacement or modification of either makes readiness and later appends fail closed.

Archive the oldest sealed segments first, and verify before removing anything:

evidence verify-audit --runtime /etc/registry-evidence/runtime.yaml
cp /var/lib/registry-evidence/audit/evidence.jsonl.00000001 "$ARCHIVE/"
rm /var/lib/registry-evidence/audit/evidence.jsonl.00000001

The evidence_audit_segments and evidence_audit_bytes gauges described in Metrics reference are the archival signal. Both are recomputed by walking the audit directory, so they fall when an operator archives sealed segments and rise again as the chain grows. Audit bytes that grow without bound are a shipper that stopped, which is the condition worth alerting on. Neither gauge observes the receiver: they report what is still on local disk, never what external storage accepted.

Registry Mint’s durable state is its governed signer configuration and public keys, the client registration files under clients.directory, and the segmented audit chain. The external Transit service retains the strict-mode private key. Onboarding, offboarding, and caller key rotation reload the client directory on SIGHUP without a restart. The client-assertion replay cache is in-memory only and clears on restart, so a restarted Registry Mint accepts a previously used jti again for whatever lifetime remains on that assertion. Keep clientAssertion.maximumLifetimeSeconds short enough that this window stays small.

Run mint verify-audit to replay the Mint chain across every retained segment. The command reports segment and record counts, the first and last sealed sequence, the chain tail, and whether the active segment was verified. A running writer keeps the active segment out of the verification pass and produces active-segment: not verified; stop Registry Mint for a complete retained-chain pass.

Relay recovery state

Relay V2 keeps no PostgreSQL-backed correctness state. It has no consultation lane, dispatch fencing, quota-bucket persistence, materialization publication history, batch-child replay, or audit-pseudonym keyring; those were Relay V1 concepts and none of them apply to Relay V2. A source is either an immutable snapshot file or a single read-only transaction against a live source, and Relay V2 keeps no separate database of its own. Its only durable, Relay-owned state is the package, the runtime file, the bound source path, and the audit sink already described in Durable state and externally retained records.

Because a package is not signed and carries no anti-rollback ratchet, Relay V2 has no rollback boundary to preserve across a recovery. Recovery is the same procedure as an ordinary deployment change: install the reviewed package, source, and runtime file at new revisioned paths and replace the complete revision. See Replace complete revisions for the exact steps.

What remains outside the product

Registry Stack does not implement a general erasure workflow for source records, audit records, or cache entries. It also does not prune operator-owned config files, source files, or old release artifacts after the product-level expiry checks above have run. Those controls belong in the operator’s storage, backup, incident-response, and data-protection processes.