Rotate credentials, keys, certificates, and trust

View as Markdown

Use this procedure to rotate a Relay cursor-integrity key or bound source path, or an Evidence Gateway signing key, audit hash secret, source credential, or source certificate authority, without exposing secret material and without widening authority. Relay V2 has no caller-key management and no configuration-signing or trust-anchor system of its own; the sections below explain why, and where that authority actually lives.

Prerequisites

Identify the material, every consumer, its current secret or trust reference, and its expiry.
Preserve a verified recovery set for the current Relay product configuration, and keep the prior Evidence Gateway package digest each rotation replaces.
Keep caller traffic outside the staged Relay instance until its checks pass; restart Evidence Gateway only after its offline checks pass first.
Use synthetic or institution-approved canaries. Do not use personal data for a rotation probe.

Ownership and trust boundary

Relay V2 owns its bound source path and its cursor-integrity key; its audit entries carry no keyed field, so it holds no audit key. It has no source credentials, no private certification authority material, no mutual TLS keys of its own, and no caller keys: a source is a read-only SQLite file Relay reads directly, and caller authority comes entirely from the configured OIDC issuer’s bearer tokens, verified against that issuer’s published keys. Evidence Gateway owns its governed public signing keys, Transit signer binding, audit hash secret, and subject-binding secret. The HMAC secrets resolve through its owner-only secret root, while the production private signing key remains in Transit (products/evidence/OPERATOR-CONTRACT.md, Secrets and keys). The deployment operator owns secret storage, certificates, the OIDC issuer Relay trusts, traffic admission, and revocation.

Relay V2 packages carry no signature and no trust anchor. relayctl package produces a package identified by an integrity digest, its package digest; relay serve re-derives that package from its inputs and refuses to start on a mismatch. There is no signing key, no anchor, no lane, and no anti-rollback ratchet to rotate for Relay V2, unlike Relay V1’s registryctl-built and registryctl-verified relay-public and relay-consultation lanes. Evidence Gateway has no registryctl-governed bundle or trust-anchor system of its own and never did: Evidence Gateway validates its governed bundle and runtime file with evidence check.

Classify the rotation

Treat suspected exposure as an incident. Revoke or disable the affected material, stop affected traffic, and preserve redacted audit evidence before routine rollout work.
Decide whether consumers need an overlap window. Caller credentials, integrity keys, and public credential-verification keys normally need overlap.
Decide whether the change remains in the secret plane or changes a governed reference. A secret-plane change retains the reviewed reference and replaces the value through the secret provider. For Relay, a governed change, a new source path, a new cursor-integrity key reference, or a new OIDC issuer binding, means authoring a new package with relayctl package and a new runtime.yaml, then replacing the complete running revision; there is no separate signing step. For Evidence Gateway, a governed change edits the bundle and runtime file, then runs evidence check before the next restart.

Operate Relay defines how Relay V2 binds deployment secrets and replaces complete revisions; Relay project authoring defines the relayctl commands that produce a package. Evidence Gateway security model traces Evidence Gateway’s secret and signing-key invariants to their tests, and products/evidence/OPERATOR-CONTRACT.md defines the incident boundary for a suspected key or audit-secret exposure.

Rotate source credentials, certificates, or source trust

Relay V2 sources have no credential, certificate, or trust-anchor boundary of their own: a source is a path to a read-only SQLite file (sources.<name>.path in runtime.yaml), pinned by the SQLite schema fingerprint the package compiled against. There is nothing to rotate in the sense Relay V1’s source connectors, private certification authority, or mutual TLS material required.

The closest V2 equivalent, replacing the file a source path points at, follows the same replace complete revisions procedure as any other deployment change:

Stage the replacement SQLite file at a new trusted, immutable path. Relay validates path components before use and refuses a symbolic link or a component writable by group or world.
Confirm the replacement file’s schema fingerprint matches what the currently reviewed package compiled against. A drifted fingerprint is a new source and needs a new package from Relay project authoring, not a runtime-only change.
Update sources.<name>.path in a new runtime file pointing at the new file.
Start a candidate on a private listener, wait for readiness, and send a smoke request through the production proxy policy.
Shift traffic, drain the previous process, and terminate it.

Private certification authority material and mutual TLS keys do not apply to Relay V2. Do not copy source certificates, keys, or destinations into the Evidence Gateway bundle: Evidence Gateway sources declare their own tlsTrustProfile bound to a separate PEM file in runtime.yaml (docs/site/src/content/docs/configure/evidence.mdx), and mixing the two namespaces will not take effect where you expect.

Rotate Evidence Gateway source credentials and source trust

Evidence Gateway source credentials live in the secret plane. A source names each credential by a secret reference in the governed bundle, and the value stays outside the package and outside every requirement’s configurationRevision, so replacing it at the same reference needs no new candidate. Whether it also needs a restart depends on the secret provider.

A secret:env/<NAME> reference reads the Evidence Gateway process’s own environment, which does not change while the process runs. Updating an orchestrator’s environment therefore reaches the process only when it restarts: drain the traffic and restart every Evidence Gateway process with the replacement value.

A secret:file/<name> reference reads an owner-only file under the runtime’s secret root, and replacing that file needs no restart. When the new bytes take effect depends on the credential kind:

basic, static-authorization, and static-api-key credentials are read from their files on every source request, so the next source request after the replacement sends the new value.
oauth2-client-credentials credentials (the client identifier, client secret, or client assertion key) are read only when Evidence Gateway exchanges them for an access token. An unexpired cached token stays in use until its bounded expiry, at most the source’s maximumCacheSeconds, and a named sourceConnections entry shares that cache across the sources that use it. When the provider must stop seeing the old token sooner, drain the traffic and restart every Evidence Gateway process that uses the credential; a restart starts with an empty cache.

Write the replacement as a new regular file and rename it over the old one, so no request reads a half-written file. The file must keep the rules every secret file follows: owned by the service identity, mode 0400 or 0600, a single hard link, and not a symbolic link. A replacement that breaks one of them makes the next read refuse the credential: the source request that needed it fails, readiness reports the source unavailable, and nothing falls back to the predecessor.

Each rename is atomic, but a credential held in more than one file is not: a basic username and password, or an OAuth client identifier with its secret or assertion key. A request or token exchange that runs between two renames reads the new value from one file and the old value from the other, and fails authentication even though both complete credentials are valid. Drain the traffic before replacing a multi-file credential, and resume it once every file is replaced.

Keep the predecessor valid at the provider until the new one has served a synthetic request; the source credential rotation reference gives the complete sequence, including emergency revocation.

A source’s TLS certificate authority is a runtime binding, not a secret. The bundle names only a logical tlsTrustProfile; runtime.yaml binds that name to a PEM file under outboundTls.trustProfiles.<name>.caBundleFile. Evidence Gateway reads that file once at startup and builds its source clients from those bytes, so replacing the file changes nothing until the next restart. Because the runtime file and its CA files stay outside the package, a new CA for the same trust profile needs no new candidate and moves no configurationRevision:

Write the new PEM bundle to the path the runtime file names, or to a new path in a new runtime.yaml, keeping it non-writable to the service identity. During a certificate-authority changeover, put both the outgoing and the incoming certificates in the bundle.
Run evidencectl doctor --runtime-config "<deployment-target>/runtime.yaml" --without-audit-lock. Doctor loads the runtime file and every CA bundle it names, so it refuses a missing, writable, or malformed replacement. --without-audit-lock leaves the audit single-writer lock to the process still serving; without it, doctor refuses while that process holds the lock. evidencectl test does not prove the CA: against an editable project it checks the compiled bundle and its fixtures, not the runtime file.
Restart each Evidence Gateway process, wait for /ready, and send one synthetic request through the source.

Changing which trust profile a source or the OIDC issuer uses, or its endpoint, secret reference, authentication policy, or audience, is a governed change instead: build and review a new candidate with evidencectl package.

Rotate caller keys

Relay V2 has no caller-key management of its own: it has no generate-api-key command, no API-key store, and no per-key revocation list. authentication.oidc in runtime.yaml names an OIDC issuer, its key source (jwksSource), audience, accepted token types, and accepted algorithms; every caller authenticates with a bearer token from that issuer, verified against the issuer’s own published keys. Caller-key rotation for Relay V2 happens entirely at the OIDC issuer: rotate the issuer’s signing keys through the issuer’s own key-rollover procedure, and rotate an individual caller’s credential (client secret, certificate, or key) through whatever mechanism that issuer or its client registry uses. Relay V2 has no compromised-key denylist of its own, unlike Evidence Gateway’s authentication.revokedKeyIds; a compromised caller credential must be revoked at the issuer, and Relay only needs a new deployment when the issuer binding itself changes (issuer, jwksSource, audience, tokenTypes, or algorithms).

Configuration signing and trust anchors do not apply to Relay V2

Relay V1 built and verified two independently signed lanes, relay-public and relay-consultation, through registryctl trust anchor rotate, registryctl trust bundle sign, registryctl trust bundle verify, and registryctl trust approved-set assemble. Relay V2 has none of this. A package built by relayctl package is identified by an integrity digest (its package digest) that relay serve re-derives and compares byte-for-byte at startup; it is not signed, has no trust anchor, no lane, no approved set, and no anti-rollback sequence. There is no anchor-rotation procedure to run, no signer key to rotate, and no bundle-verification command to invoke for Relay V2: replacing a package is the same replace complete revisions procedure as any other deployment change, and the only integrity check is the automatic startup re-derivation every relay serve performs.

Rotate Evidence Gateway signing material

Evidence Gateway’s signing key rotation is a rehearsed procedure, not new material for this page: create the replacement non-exportable P-256 Transit key version, convert its public PEM with evidencectl jwk from-pem and publish the resulting public JWK, then switch signing.activePublicJwkFile and the pinned signer version in the reviewed bundle before running evidence check and restarting. Rotate Evidence Gateway signing keys walks through each step, and products/evidence/OPERATOR-CONTRACT.md (Secrets and keys) is the binding contract behind it. Missing or failed signing is fail-closed: a rotation mistake surfaces as refused requests, never as an unsigned assertion.

Audit hash-secret rotation is a separate event from signing-key rotation. Generate a fresh master into a new secret file, build a governed bundle revision whose audit.hashKeyRef names it and whose audit.hashKeyVersion is one higher, run evidence check, and restart. Every pseudonym carries its version, so the same audit path serves both versions. Never replace the master bytes without raising the version: the runtime cannot detect that, and it silently changes every pseudonym under an unchanged prefix. Retain the old master under the audit retention policy for as long as pseudonyms from its version must be recomputable. Rotate the audit master gives the steps.

Expected evidence

Retain:

The relayctl check (and relayctl check --production) report for a Relay V2 project change, and the package digest for the staged candidate.
The evidence check output and the readiness result for a rotated Evidence Gateway signing key.
Health and readiness results from the staged Relay instance; Relay V2 has no redacted posture report to retain alongside them.
A synthetic or authorized bounded canary result.
The old-material retirement time and the approved overlap window.

Evidence Gateway records may contain key ids, product ids, scopes, and certificate metadata. They must not contain raw keys, fingerprints, private certificates, tokens, environment values, or full configuration dumps.

What this proves

relayctl check proves that a Relay V2 project’s compiled contract is internally consistent and, under --production, that it satisfies the stricter production posture; it does not prove the deployed runtime binding is correct. relay serve’s startup re-derivation proves that the installed package’s bytes match what relayctl package produced from the reviewed inputs; there is no separate operator-invoked bundle-verification command the way Relay V1’s registryctl trust bundle verify was. evidence check proves Evidence Gateway’s key material and configuration are internally consistent before a restart. Relay and Evidence Gateway share no signing or activation system. Staged readiness and a bounded canary prove the tested runtime path.

These gates do not prove country approval, legal authority, every source operation, every caller migration, remote audit retention, or live interoperability that was not tested.

Roll back or recover

Before the replacement serves traffic, restore the prior secret reference, package, source, and runtime file. Relay V2 has no anti-rollback state and no configuration sequence to preserve: recovery is installing the prior reviewed package, source, and runtime file and replacing the complete revision again, the same procedure as any other rollback.

Keep old Evidence Gateway public verification keys published in the JWKS while assertions signed by those keys can still be verified: at least the maximum assertion validity plus allowed clock skew (products/evidence/OPERATOR-CONTRACT.md, Secrets and keys).

Escalate

Escalate to the product security owner when material may be exposed, a private key or token reaches logs, a Relay OIDC issuer binding cannot be rotated without an outage, an old Evidence Gateway public key cannot remain published for its required retention window, or a canary requires live country data.