Rotate credentials, keys, certificates, and trust

View as Markdown

Use this procedure to rotate a Relay audit-integrity key, cursor-integrity key, or bound source path, an Evidence Gateway signing key, or a Registry Mint signing key or client key, without exposing secret material and without widening authority. Relay V2 has no caller-key management and no configuration-signing or trust-anchor system of its own; the sections below explain why, and where that authority actually lives.

Prerequisites

Identify the material, every consumer, its current secret or trust reference, and its expiry.
Preserve a verified recovery set for the current Relay product configuration, and keep the prior Evidence Gateway bundle revision and Registry Mint configuration file each rotation replaces.
Keep caller traffic outside the staged Relay instance until its checks pass; restart Evidence Gateway and Registry Mint only after their own offline checks pass first.
Use synthetic or institution-approved canaries. Do not use personal data for a rotation probe.

Ownership and trust boundary

Relay V2 owns its bound source path, its audit-integrity key, and its cursor-integrity key. It has no source credentials, no private certification authority material, no mutual TLS keys of its own, and no caller keys: a source is a read-only SQLite file Relay reads directly, and caller authority comes entirely from the configured OIDC issuer’s bearer tokens, verified against that issuer’s published keys. Evidence Gateway owns its governed public signing keys, Transit signer binding, audit hash secret, and subject-binding secret. The HMAC secrets resolve through its owner-only secret root, while the production private signing key remains in Transit (products/evidence/OPERATOR-CONTRACT.md, Secrets and keys). Registry Mint owns its governed public signing keys, Transit signer binding, audit hash secret, and client registry. Each registered client has one or more public keys and one granted authority. The deployment operator owns secret storage, certificates, the OIDC issuer Relay trusts, traffic admission, and revocation.

Relay V2 packages carry no signature and no trust anchor. relayctl package produces a package identified by an integrity digest, packageRevision; relay serve re-derives that package from its inputs and refuses to start on a mismatch. There is no signing key, no anchor, no lane, and no anti-rollback ratchet to rotate for Relay V2, unlike Relay V1’s registryctl-built and registryctl-verified relay-public and relay-consultation lanes. Evidence Gateway and Registry Mint have no registryctl-governed bundle or trust-anchor system of their own, and never did: Evidence Gateway validates its governed bundle and runtime file with evidence check, and Registry Mint validates its configuration with mint check.

Classify the rotation

Treat suspected exposure as an incident. Revoke or disable the affected material, stop affected traffic, and preserve redacted audit evidence before routine rollout work.
Decide whether consumers need an overlap window. Caller credentials, integrity keys, and public credential-verification keys normally need overlap.
Decide whether the change remains in the secret plane or changes a governed reference. A secret-plane change retains the reviewed reference and replaces the value through the secret provider. For Relay, a governed change, a new source path, a new audit- or cursor-integrity key reference, or a new OIDC issuer binding, means authoring a new package with relayctl package and a new runtime.yaml, then replacing the complete running revision; there is no separate signing step. For Evidence Gateway and Registry Mint, a governed change edits the bundle or configuration file directly, then runs evidence check or mint check before the next restart.

Operate Relay defines how Relay V2 binds deployment secrets and replaces complete revisions; Relay project authoring defines the relayctl commands that produce a package. Evidence Gateway security model traces Evidence Gateway’s secret and signing-key invariants to their tests, and products/evidence/OPERATOR-CONTRACT.md defines the incident boundary for a suspected key or audit-secret exposure.

Rotate source credentials, certificates, or source trust

Relay V2 sources have no credential, certificate, or trust-anchor boundary of their own: a source is a path to a read-only SQLite file (sources.<name>.path in runtime.yaml), pinned by the SQLite schema fingerprint the package compiled against. There is nothing to rotate in the sense Relay V1’s source connectors, private certification authority, or mutual TLS material required.

The closest V2 equivalent, replacing the file a source path points at, follows the same replace complete revisions procedure as any other deployment change:

Stage the replacement SQLite file at a new trusted, immutable path. Relay validates path components before use and refuses a symbolic link or a component writable by group or world.
Confirm the replacement file’s schema fingerprint matches what the currently reviewed package compiled against. A drifted fingerprint is a new source and needs a new package from Relay project authoring, not a runtime-only change.
Update sources.<name>.path in a new runtime file pointing at the new file.
Start a candidate on a private listener, wait for readiness, and send a smoke request through the production proxy policy.
Shift traffic, drain the previous process, and terminate it.

Private certification authority material and mutual TLS keys do not apply to Relay V2. Do not copy source certificates, keys, or destinations into the Evidence Gateway bundle: Evidence Gateway sources declare their own tlsTrustProfile bound to a separate PEM file in runtime.yaml (docs/site/src/content/docs/configure/evidence.mdx), and mixing the two namespaces will not take effect where you expect.

Rotate caller keys

Relay V2 has no caller-key management of its own: it has no generate-api-key command, no API-key store, and no per-key revocation list. authentication.issuer in runtime.yaml names an OIDC issuer, its discovery URL, audience, accepted token types, and accepted algorithms; every caller authenticates with a bearer token from that issuer, verified against the issuer’s own published keys. Caller-key rotation for Relay V2 happens entirely at the OIDC issuer: rotate the issuer’s signing keys through the issuer’s own key-rollover procedure, and rotate an individual caller’s credential (client secret, certificate, or key) through whatever mechanism that issuer or its client registry uses. Relay V2 has no compromised-key denylist of its own, unlike Evidence Gateway’s authentication.revokedKeyIds; a compromised caller credential must be revoked at the issuer, and Relay only needs a new deployment when the issuer binding itself changes (id, discoveryUrl, audience, tokenTypes, or algorithms).

For a Registry Mint client key, add or update the client’s registration file under clients.directory with the new public JWK, keep the old registration in place during the overlap window, then send the running Registry Mint process SIGHUP to reload the client registry without a restart (docs/site/src/content/docs/configure/mint.mdx). Registry Mint reloads the whole client registry atomically: a malformed replacement file fails the reload and Registry Mint keeps serving the previous registry rather than a partial one. Remove the old registration only after every client has moved to the new key and the configured maximum client-assertion lifetime plus 30 seconds has elapsed. A compromised client key is removed immediately and reloaded without an overlap window.

Rotating Registry Mint’s own signing key is a configuration change, not a client-registry reload: publish the next public JWK, then deploy and restart every replica so all of them publish the overlap set. In a second candidate, switch signing.activePublicJwkFile and the pinned Transit key version together, keep the old public JWK in signing.publishedPublicJwkFiles, then deploy and restart every replica again. Raise the Transit key’s min_encryption_version and remove the old public JWK only after the maximum token lifetime plus consumer skew has elapsed.

Evidence Gateway does not need reconfiguration for a planned Registry Mint rotation. It fetches Registry Mint’s JWKS from the configured jwksUri on its own cache lifecycle, provided both services already agree on issuer and accessTokens.claims. For a compromised Mint service key, add its thumbprint to Evidence Gateway authentication.revokedKeyIds in the same incident rollout and restart every affected consumer. The denylist takes precedence over a cached JWKS (products/evidence/OPERATOR-CONTRACT.md, Startup and readiness).

Configuration signing and trust anchors do not apply to Relay V2

Relay V1 built and verified two independently signed lanes, relay-public and relay-consultation, through registryctl trust anchor rotate, registryctl trust bundle sign, registryctl trust bundle verify, and registryctl trust approved-set assemble. Relay V2 has none of this. A package built by relayctl package is identified by an integrity digest (packageRevision) that relay serve re-derives and compares byte-for-byte at startup; it is not signed, has no trust anchor, no lane, no approved set, and no anti-rollback sequence. There is no anchor-rotation procedure to run, no signer key to rotate, and no bundle-verification command to invoke for Relay V2: replacing a package is the same replace complete revisions procedure as any other deployment change, and the only integrity check is the automatic startup re-derivation every relay serve performs.

Rotate Evidence Gateway and Registry Mint signing material

Evidence Gateway’s signing key rotation is a rehearsed procedure, not new material for this page: create the replacement non-exportable P-256 Transit key version, publish its public JWK, then switch signing.activePublicJwkFile and the pinned signer version in the reviewed bundle before running evidence check and restarting. Rotate Evidence Gateway signing keys walks through each step, and products/evidence/OPERATOR-CONTRACT.md (Secrets and keys) is the binding contract behind it. Missing or failed signing is fail-closed: a rotation mistake surfaces as refused requests, never as an unsigned assertion.

Registry Mint’s signing-key rotation is covered above, under Rotate caller keys, next to the client-key rotation it is usually done alongside.

Audit hash-secret rotation is a separate event from signing-key rotation. It begins a new epoch: drain and stop, verify and record the old head, archive the old runtime, master, segments, and head, then use a fresh path, a fresh master, and incremented hashKeyVersion. Retain the old master under the audit retention policy because it verifies the archived epoch and its pseudonyms.

Expected evidence

Retain:

The relayctl check (and relayctl check --production) report for a Relay V2 project change, and the package’s packageRevision digest for the staged candidate.
The evidence check output and the readiness result for a rotated Evidence Gateway signing key.
The mint check output for a rotated Registry Mint signing key or client registration.
Health and readiness results from the staged Relay instance; Relay V2 has no redacted posture report to retain alongside them.
A synthetic or authorized bounded canary result.
The old-material retirement time and the approved overlap window.

Evidence Gateway records may contain key ids, product ids, scopes, and certificate metadata. They must not contain raw keys, fingerprints, private certificates, tokens, environment values, or full configuration dumps.

What this proves

relayctl check proves that a Relay V2 project’s compiled contract is internally consistent and, under --production, that it satisfies the stricter production posture; it does not prove the deployed runtime binding is correct. relay serve’s startup re-derivation proves that the installed package’s bytes match what relayctl package produced from the reviewed inputs; there is no separate operator-invoked bundle-verification command the way Relay V1’s registryctl trust bundle verify was. evidence check and mint check prove their own deployment’s key material and configuration are internally consistent before a restart; neither proves the other product’s trust lineage, because Relay, Evidence Gateway, and Registry Mint share no signing or activation system. Staged readiness and a bounded canary prove the tested runtime path.

These gates do not prove country approval, legal authority, every source operation, every caller migration, remote audit retention, or live interoperability that was not tested.

Roll back or recover

Before the replacement serves traffic, restore the prior secret reference, package, source, and runtime file. Relay V2 has no anti-rollback state and no configuration sequence to preserve: recovery is installing the prior reviewed package, source, and runtime file and replacing the complete revision again, the same procedure as any other rollback.

Keep old Evidence Gateway public verification keys published in the JWKS while assertions signed by those keys can still be verified: at least the maximum assertion validity plus allowed clock skew (products/evidence/OPERATOR-CONTRACT.md, Secrets and keys). Keep Registry Mint staged and Evidence Gateway’s traffic blocked when Registry Mint’s issuer, audience, or claim names do not exactly match Evidence Gateway’s configured authentication profile: a mismatch fails closed as auth.invalid_credential rather than granting the wrong authority (docs/site/src/content/docs/configure/mint.mdx, Troubleshooting).

Escalate

Escalate to the product security owner when material may be exposed, a private key or token reaches logs, a Relay OIDC issuer binding cannot be rotated without an outage, an old Evidence Gateway public key cannot remain published for its required retention window, or a canary requires live country data.