Registry stack documentation: machine-readable Markdown.
Index of all pages: https://docs.registrystack.org/v/0.38.0/llms.txt
Full corpus: https://docs.registrystack.org/v/0.38.0/llms-full.txt

# Rotate credentials, keys, certificates, and trust

> Rotate Registry Relay and Evidence Gateway credentials, signing keys, and trust material without exposing secret material or widening authority.

Use this procedure to rotate a Relay cursor-integrity key or bound source path, or an Evidence
Gateway signing key, audit hash secret, source credential, or source certificate authority, without
exposing secret material and without widening authority. Relay V2 has no caller-key management and
no configuration-signing or trust-anchor system of its own; the sections below explain why, and
where that authority actually lives.

## Prerequisites

- Identify the material, every consumer, its current secret or trust reference, and its expiry.
- Preserve a verified recovery set for the current Relay product configuration, and keep the
  prior Evidence Gateway package digest each rotation replaces.
- Keep caller traffic outside the staged Relay instance until its checks pass; restart Evidence Gateway
  only after its offline checks pass first.
- Use synthetic or institution-approved canaries. Do not use personal data for a rotation probe.

## Ownership and trust boundary

Relay V2 owns its bound source path and its cursor-integrity key; its audit entries carry no keyed
field, so it holds no audit key. It has
no source credentials, no private certification authority material, no mutual TLS keys of its own,
and no caller keys: a source is a read-only SQLite file Relay reads directly, and caller authority
comes entirely from the configured OIDC issuer's bearer tokens, verified against that issuer's
published keys.
Evidence Gateway owns its governed public signing keys, Transit signer binding, audit hash secret,
and subject-binding secret. The HMAC secrets resolve through its owner-only secret root, while the
production private signing key remains in Transit (`products/evidence/OPERATOR-CONTRACT.md`, Secrets
and keys).
The deployment operator owns secret storage, certificates, the OIDC issuer Relay trusts, traffic
admission, and revocation.

Relay V2 packages carry no signature and no trust anchor. `relayctl package` produces a package
identified by an integrity digest, its package digest; `relay serve` re-derives that package from
its inputs and refuses to start on a mismatch. There is no signing key, no anchor, no lane, and no
anti-rollback ratchet to rotate for Relay V2, unlike Relay V1's `registryctl`-built and
`registryctl`-verified `relay-public` and `relay-consultation` lanes.
Evidence Gateway has no `registryctl`-governed bundle or trust-anchor system of its own and never
did: Evidence Gateway validates its governed bundle and runtime file with `evidence check`.

{/* Evidence: crates/registry-relay-v2/src/contract.rs, crates/registry-relay-v2/src/startup.rs,
    crates/registry-relayctl/src/lib.rs, products/relay-v2/CONCEPT.md,
    products/evidence/OPERATOR-CONTRACT.md,
    docs/site/src/content/docs/tutorials/rotate-evidence-signing-keys.mdx. */}

## Classify the rotation

1. Treat suspected exposure as an incident.
   Revoke or disable the affected material, stop affected traffic, and preserve redacted audit
   evidence before routine rollout work.
2. Decide whether consumers need an overlap window.
   Caller credentials, integrity keys, and public credential-verification keys normally need
   overlap.
3. Decide whether the change remains in the secret plane or changes a governed reference.
   A secret-plane change retains the reviewed reference and replaces the value through the
   secret provider.
   For Relay, a governed change, a new source path, a new cursor-integrity key reference, or a
   new OIDC issuer binding, means authoring a new package with `relayctl package`
   and a new `runtime.yaml`, then replacing the complete running revision; there is no separate
   signing step. For Evidence Gateway, a governed change edits the bundle and runtime file, then runs
   `evidence check` before the next restart.

[Operate Relay](../../relay/) defines how Relay V2 binds deployment secrets and replaces complete
revisions; [Relay project authoring](../../../configure/relay/) defines the `relayctl` commands
that produce a package.
[Evidence Gateway security model](../../../security/evidence/) traces Evidence Gateway's secret and signing-key
invariants to their tests, and `products/evidence/OPERATOR-CONTRACT.md` defines the incident
boundary for a suspected key or audit-secret exposure.

## Rotate source credentials, certificates, or source trust

Relay V2 sources have no credential, certificate, or trust-anchor boundary of their own: a source
is a path to a read-only SQLite file (`sources.<name>.path` in `runtime.yaml`), pinned by the
SQLite schema fingerprint the package compiled against. There is nothing to rotate in the sense
Relay V1's source connectors, private certification authority, or mutual TLS material required.

The closest V2 equivalent, replacing the file a source path points at, follows the same
[replace complete revisions](../../relay/#replace-complete-revisions) procedure as any other
deployment change:

1. Stage the replacement SQLite file at a new trusted, immutable path. Relay validates path
   components before use and refuses a symbolic link or a component writable by group or world.
2. Confirm the replacement file's schema fingerprint matches what the currently reviewed package
   compiled against. A drifted fingerprint is a new source and needs a new package from
   [Relay project authoring](../../../configure/relay/), not a runtime-only change.
3. Update `sources.<name>.path` in a new runtime file pointing at the new file.
4. Start a candidate on a private listener, wait for readiness, and send a smoke request through
   the production proxy policy.
5. Shift traffic, drain the previous process, and terminate it.

Private certification authority material and mutual TLS keys do not apply to Relay V2. Do not copy
source certificates, keys, or destinations into the Evidence Gateway bundle: Evidence Gateway sources
declare their own `tlsTrustProfile` bound to a separate PEM file in `runtime.yaml`
(`docs/site/src/content/docs/configure/evidence.mdx`), and mixing the two namespaces will not take
effect where you expect.

## Rotate Evidence Gateway source credentials and source trust

Evidence Gateway source credentials live in the secret plane. A source names each credential by a
secret reference in the governed bundle, and the value stays outside the package and outside every
requirement's `configurationRevision`, so replacing it at the same reference needs no new candidate.
Whether it also needs a restart depends on the secret provider.

A `secret:env/<NAME>` reference reads the Evidence Gateway process's own environment, which does not
change while the process runs. Updating an orchestrator's environment therefore reaches the process
only when it restarts: drain the traffic and restart every Evidence Gateway process with the
replacement value.

A `secret:file/<name>` reference reads an owner-only file under the runtime's secret root, and
replacing that file needs no restart. When the new bytes take effect depends on the credential kind:

- **`basic`, `static-authorization`, and `static-api-key` credentials** are read from their files on
  every source request, so the next source request after the replacement sends the new value.
- **`oauth2-client-credentials` credentials** (the client identifier, client secret, or client
  assertion key) are read only when Evidence Gateway exchanges them for an access token. An
  unexpired cached token stays in use until its bounded expiry, at most the source's
  `maximumCacheSeconds`, and a named `sourceConnections` entry shares that cache across the sources
  that use it. When the provider must stop seeing the old token sooner, drain the traffic and restart
  every Evidence Gateway process that uses the credential; a restart starts with an empty cache.

Write the replacement as a new regular file and rename it over the old one, so no request reads a
half-written file. The file must keep the rules every secret file follows: owned by the service
identity, mode `0400` or `0600`, a single hard link, and not a symbolic link. A replacement that
breaks one of them makes the next read refuse the credential: the source request that needed it
fails, readiness reports the source unavailable, and nothing falls back to the predecessor.

Each rename is atomic, but a credential held in more than one file is not: a `basic` username and
password, or an OAuth client identifier with its secret or assertion key. A request or token
exchange that runs between two renames reads the new value from one file and the old value from
the other, and fails authentication even though both complete credentials are valid. Drain the
traffic before replacing a multi-file credential, and resume it once every file is replaced.

Keep the predecessor valid at the provider until the new one has served a synthetic request; the
[source credential rotation](../../../products/registry-evidence/source-credential-rotation/)
reference gives the complete sequence, including emergency revocation.

A source's TLS certificate authority is a runtime binding, not a secret. The bundle names only a
logical `tlsTrustProfile`; `runtime.yaml` binds that name to a PEM file under
`outboundTls.trustProfiles.<name>.caBundleFile`. Evidence Gateway reads that file once at startup
and builds its source clients from those bytes, so replacing the file changes nothing until the
next restart. Because the runtime file and its CA files stay outside the package, a new CA for the
same trust profile needs no new candidate and moves no `configurationRevision`:

1. Write the new PEM bundle to the path the runtime file names, or to a new path in a new
   `runtime.yaml`, keeping it non-writable to the service identity. During a certificate-authority
   changeover, put both the outgoing and the incoming certificates in the bundle.
2. Run `evidencectl doctor --runtime-config "<deployment-target>/runtime.yaml" --without-audit-lock`.
   Doctor loads the runtime file and every CA bundle it names, so it refuses a missing, writable, or
   malformed replacement. `--without-audit-lock` leaves the audit single-writer lock to the process
   still serving; without it, doctor refuses while that process holds the lock. `evidencectl test`
   does not prove the CA: against an editable project it checks the compiled bundle and its fixtures,
   not the runtime file.
3. Restart each Evidence Gateway process, wait for `/ready`, and send one synthetic request through
   the source.

Changing which trust profile a source or the OIDC issuer uses, or its endpoint, secret reference,
authentication policy, or audience, is a governed change instead: build and review a new candidate
with `evidencectl package`.

{/* Evidence: crates/registry-evidence/src/source.rs AuthenticationPlan access_token maximum_cache_lifetime;
    crates/registry-platform-config/src/secrets.rs read_secret_file validate_file_metadata read_environment;
    crates/registry-evidencectl/src/runtime.rs without_audit_lock;
    crates/registry-evidencectl/src/fixtures.rs TargetedEditable;
    crates/registry-evidence/src/bundle.rs RuntimeDocument ca_bundles;
    products/evidence/reference/request-adapter/deployment-projects/SOURCE-CREDENTIAL-ROTATION.md;
    products/evidence/reference/request-adapter/deployment-projects/CONFIG.md caBundleFile. */}

## Rotate caller keys

Relay V2 has no caller-key management of its own: it has no `generate-api-key` command, no
API-key store, and no per-key revocation list. `authentication.oidc` in `runtime.yaml` names an
OIDC issuer, its key source (`jwksSource`), audience, accepted token types, and accepted algorithms; every
caller authenticates with a bearer token from that issuer, verified against the issuer's own
published keys. Caller-key rotation for Relay V2 happens entirely at the OIDC issuer: rotate the
issuer's signing keys through the issuer's own key-rollover procedure, and rotate an individual
caller's credential (client secret, certificate, or key) through whatever mechanism that issuer or
its client registry uses. Relay V2 has no compromised-key denylist of its own, unlike Evidence
Gateway's `authentication.revokedKeyIds`; a compromised caller credential must be revoked at the
issuer, and Relay only needs a new deployment when the issuer binding itself changes (`issuer`,
`jwksSource`, `audience`, `tokenTypes`, or `algorithms`).

## Configuration signing and trust anchors do not apply to Relay V2

Relay V1 built and verified two independently signed lanes, `relay-public` and
`relay-consultation`, through `registryctl trust anchor rotate`, `registryctl trust bundle sign`,
`registryctl trust bundle verify`, and `registryctl trust approved-set assemble`. Relay V2 has none
of this. A package built by `relayctl package` is identified by an integrity digest
(its package digest) that `relay serve` re-derives and compares byte-for-byte at startup; it is not
signed, has no trust anchor, no lane, no approved set, and no anti-rollback sequence. There is no
anchor-rotation procedure to run, no signer key to rotate, and no bundle-verification command to
invoke for Relay V2: replacing a package is the same
[replace complete revisions](../../relay/#replace-complete-revisions) procedure as any other
deployment change, and the only integrity check is the automatic startup re-derivation every
`relay serve` performs.

## Rotate Evidence Gateway signing material

Evidence Gateway's signing key rotation is a rehearsed procedure, not new material for this page:
create the replacement non-exportable P-256 Transit key version, convert its public PEM with
`evidencectl jwk from-pem` and publish the resulting public JWK, then
switch `signing.activePublicJwkFile` and the pinned signer version in the reviewed bundle before
running `evidence check` and restarting.
[Rotate Evidence Gateway signing keys](../../../tutorials/rotate-evidence-signing-keys/)
walks through each step, and `products/evidence/OPERATOR-CONTRACT.md` (Secrets and keys) is the
binding contract behind it.
Missing or failed signing is fail-closed: a rotation mistake surfaces as refused requests, never
as an unsigned assertion.

Audit hash-secret rotation is a separate event from signing-key rotation. Generate a fresh master
into a new secret file, build a governed bundle revision whose `audit.hashKeyRef` names it and
whose `audit.hashKeyVersion` is one higher, run `evidence check`, and restart. Every pseudonym
carries its version, so the same audit path serves both versions. Never replace the master bytes
without raising the version: the runtime cannot detect that, and it silently changes every
pseudonym under an unchanged prefix. Retain the old master under the audit retention policy for as
long as pseudonyms from its version must be recomputable.
[Rotate the audit master](../../evidence-audit/#rotate-the-audit-master) gives the steps.

## Expected evidence

Retain:

- The `relayctl check` (and `relayctl check --production`) report for a Relay V2 project change,
  and the package digest for the staged candidate.
- The `evidence check` output and the readiness result for a rotated Evidence Gateway signing key.
- Health and readiness results from the staged Relay instance; Relay V2 has no redacted posture
  report to retain alongside them.
- A synthetic or authorized bounded canary result.
- The old-material retirement time and the approved overlap window.

Evidence Gateway records may contain key ids, product ids, scopes, and certificate metadata.
They must not contain raw keys, fingerprints, private certificates, tokens, environment values, or
full configuration dumps.

## What this proves

`relayctl check` proves that a Relay V2 project's compiled contract is internally consistent and,
under `--production`, that it satisfies the stricter production posture; it does not prove the
deployed runtime binding is correct. `relay serve`'s startup re-derivation proves that the
installed package's bytes match what `relayctl package` produced from the reviewed inputs; there is
no separate operator-invoked bundle-verification command the way Relay V1's `registryctl trust
bundle verify` was.
`evidence check` proves Evidence Gateway's key material and configuration are internally consistent
before a restart. Relay and Evidence Gateway share no signing or activation system.
Staged readiness and a bounded canary prove the tested runtime path.

These gates do not prove country approval, legal authority, every source operation, every caller
migration, remote audit retention, or live interoperability that was not tested.

## Roll back or recover

Before the replacement serves traffic, restore the prior secret reference, package, source, and
runtime file. Relay V2 has no anti-rollback state and no configuration sequence to preserve:
recovery is installing the prior reviewed package, source, and runtime file and replacing the
complete revision again, the same procedure as any other rollback.

Keep old Evidence Gateway public verification keys published in the JWKS while assertions signed by those
keys can still be verified: at least the maximum assertion validity plus allowed clock skew
(`products/evidence/OPERATOR-CONTRACT.md`, Secrets and keys).

## Escalate

Escalate to the product security owner when material may be exposed, a private key or token
reaches logs, a Relay OIDC issuer binding cannot be rotated without an outage, an old Evidence
Gateway public key cannot remain published for its required retention window, or a canary requires
live country data.

## Next

- [Relay project authoring](../../../configure/relay/)
- [Operate Relay](../../relay/)
- [Inspect and diagnose a running deployment](../inspect-and-diagnose/)
- [Configure Transit signing for Evidence Gateway](../../../tutorials/move-evidence-to-production-signing/)