Rotate credentials and trust

View as Markdown

If you operate Registry Evidence, Base Registry Engine (BReg), or Registry Casework, rotate credentials and trust in the boundary that owns them. Keep each product’s configured authority and recovery context intact while you replace the credential.

Prerequisites

Preserve a verified recovery copy of the active configuration and referenced public keys.
Record the active configuration revision and the credentials being replaced.
Prepare the replacement secret in the operator’s secret store without writing it into governed configuration or logs.
Choose an overlap window that covers in-flight requests and the documented verification period.

Classify the rotation

Decide whether the change affects caller authentication, source access, transport trust, assertion signing, or several boundaries. Treat each boundary as a separate review item even when one deployment revision carries them together.

Rotate source credentials or trust

Evidence source credentials live in the secret plane. The governed bundle names them by secret reference, so replacing a value at the same reference does not change the package or a requirement’s configurationRevision. Restart behavior depends on the provider and credential kind.

secret:env/<NAME> reads the process environment. Restart and drain every process to apply an orchestrator environment change.
File-backed basic, static-authorization, and static-api-key credentials are read on every source request. The next request after atomic file replacement uses the new value.
File-backed oauth2-client-credentials and oauth2-private-key-jwt values are read when Evidence exchanges them for an access token. An unexpired token remains cached for at most maximumCacheSeconds, and a named sourceConnections entry shares that cache among its sources. Drain and restart every affected process when the old token must stop sooner; restart begins with an empty cache.

Write replacement bytes to a new regular file and rename it over the old file. Keep the file owned by the service identity, mode 0400 or 0600, with one hard link and no symbolic link. A failed metadata check makes the next source request refuse the credential, makes readiness report the source unavailable, and never falls back to the predecessor.

A credential spanning several files is not replaced atomically as a unit. Drain traffic before replacing a Basic username and password or an OAuth client identifier and secret or assertion key, then resume only after all files are in place. Keep the predecessor valid until the new credential has served a synthetic request.

A source CA bundle is a runtime binding, not a secret. Evidence reads outboundTls.trustProfiles.<name>.caBundleFile once at startup. Replacing the file therefore needs a restart, but no new candidate when the logical trust profile is unchanged:

Write a non-writable PEM bundle to the named path. During CA rollover, include both outgoing and incoming certificates.
Run evidencectl doctor --runtime-config "<deployment-target>/runtime.yaml" --without-audit-lock. Doctor validates the runtime file and CA bundles without contending for the serving process’s audit lock. evidencectl test checks an editable project and does not prove the runtime CA.
Restart each process, wait for /ready, and send one synthetic request through the source.

Changing a source endpoint, authentication policy, secret reference, trust-profile selection, audience, or OIDC issuer binding is governed. Build and review a new candidate.

Rotate caller trust

Coordinate issuer signing-key changes with the issuer’s published key set and cache lifetime. Test tokens signed by the new key before activation, then confirm expired or revoked credentials fail. Changing issuer, audience, algorithm, or entitlement rules is an authorization change and requires a reviewed configuration revision.

Rotate Registry Evidence signing material

Create the replacement non-exportable P-256 Transit key version, convert its public PEM with evidencectl jwk from-pem, and publish the public JWK. Then switch signing.activePublicJwkFile and the pinned signer version in a reviewed bundle, run evidence check --require-runtime-dependencies on the target host, restart, and verify a stored assertion against independently retained expectations. Retain the prior public key for the maximum assertion validity plus allowed clock skew. Missing or failed signing is fail-closed; it never produces an unsigned assertion.

Rotate Registry Evidence signing keys provides the complete sequence.

Audit hash-master rotation is separate. Generate a fresh master in a new secret file and build a bundle revision whose audit.hashKeyRef names it and whose audit.hashKeyVersion is one higher. Run evidence check --require-runtime-dependencies on the target host and restart. Every pseudonym records its version, allowing both versions in one audit path. Never replace the bytes without raising the version because the runtime cannot detect that substitution and the unchanged prefix would silently identify different pseudonyms. Retain the old master as long as its pseudonyms must remain recomputable. See Rotate the audit master.

Rotate BReg or Casework database credentials

Both services resolve database secret references when they create their connection pools. Replacing the secret file does not refresh an existing pool.

Drain traffic and reconcile uncertain writes using their original caller, body, and idempotency key before changing credentials.
Change the password for the existing runtime database role through your protected database administration session. Update its connection URL secret atomically, preserving file ownership and mode. Keep the migration role separate; rotate its secret only when rotating that role.

Run the matching preflight with the replacement runtime credential:

bregctl doctor --runtime-config /etc/breg/runtime.yaml
caseworkctl doctor --runtime-config /etc/registry-casework/runtime.yaml

Restart every affected service process. Check /ready, then perform one authorized synthetic read and mutation through the normal client. Confirm the old credential cannot open a fresh database connection before returning traffic.

A failed preflight means the replacement secret, transport trust, or grants are not ready. Keep traffic drained while you repair them. A new database role needs the product’s runtime grants; changing a password on the existing role preserves those grants. For Casework, casework.activation.ledger-unreadable after a role change requires an operator apply with the migration credential to restore runtime grants, followed by plan and doctor.

Rotate BReg and Casework caller or service trust

For discovery or URI-backed issuer keys, publish the replacement public key before issuing tokens with it and account for the configured key-cache lifetime. For a static key set, replace the referenced document and restart every serving instance. Test both a valid replacement token and an invalid token against the configured issuer, audience, client restrictions, and profile.

Treat an issuer or principal-mapping change as a separate authorization migration. BReg retry identity includes the issuer and principal; Casework Directory membership and prior review decisions also use qualified identities. Resolve pending work before changing that mapping, rather than retrying a lost write as a new caller.

Replace Casework’s configured BReg service credential or completion-delivery credential in the secret store and restart Casework to rebuild its clients. The human context path must continue to use the human caller’s own token and source profile. Run BReg and Casework at the same Registry Stack version; source.version-mismatch requires a matching deployment, not broader source permissions.

Keep audit-key rotation separate from database or caller credentials. Changing a pseudonymization key changes future audit references; retain old keys under your archive policy when you need to recompute earlier references. Changing BReg’s cursor secret invalidates existing cursors, so clients must restart their queries. For encrypted BReg fields, follow the dedicated field-encryption custody guidance before changing Transit keys.

Expected evidence

Keep the matching product preflight report, readiness result, activation time, synthetic request results, audit records, and old-credential refusal result. For signing-key rotation, also retain the old and new public key identifiers and revocation evidence. Never record private keys, bearer tokens, or source credentials.

Roll back or recover

Restore the previous reviewed configuration only while its credentials remain valid and its use is authorized. If the old credential was revoked because of compromise, fix forward with a new credential and configuration revision.