Rotate credentials, keys, certificates, and trust

View as Markdown

Use this procedure to rotate a source credential, caller key, Notary-to-Relay workload token, configuration signer, credential-issuance signer, certificate, or trust anchor without exposing secret material or widening authority.

Prerequisites

Identify the material, every consumer, its current secret or trust reference, and its expiry.
Preserve a verified recovery set for the current Relay and Notary product configurations.
Keep caller traffic outside the staged instances until the affected product checks pass.
Use synthetic or institution-approved canaries. Do not use personal data for a rotation probe.

Ownership and trust boundary

Registry Relay owns source destinations, source credentials, private certification authority material, mutual TLS keys, source protocol credentials, and Relay caller keys. Registry Notary owns its Relay workload token, Notary caller policy, and evidence-signing keys. The deployment operator owns secret storage, certificates, product trust anchors, traffic admission, and revocation.

Configuration signing keys and trust anchors are product-specific. Relay and Notary bundles have separate signatures, anchors, anti-rollback state, and activation events. A combined project has no signed project-root bundle or atomic activation coordinator.

Classify the rotation

Treat suspected exposure as an incident. Revoke or disable the affected material, stop affected traffic, and preserve redacted audit evidence before routine rollout work.
Decide whether consumers need an overlap window. Caller keys, certificate chains, configuration signers, and public credential-verification keys normally need overlap.
Decide whether the change remains in the secret plane or changes a governed reference. A secret-plane change retains the reviewed reference and replaces the value through the secret provider. A governed change produces a reviewed product configuration and a new signed product bundle.

The Notary deployment hardening runbook defines the incident boundary. The Relay operations runbook defines Relay API-key and runtime-secret rotation.

Rotate source credentials, certificates, or source trust

Add the replacement to the operator-owned Relay secret and trust boundary. Use a versioned reference when the provider treats references as immutable.
Change only the affected Relay environment binding. Do not change the source origin, method, path rules, projection, or limits as part of a credential rotation.

Run the offline environment gate:

registryctl preflight \
  --project-dir <project> \
  --environment <environment> \
  --format json

Compare the change with the verified Relay baseline, then build, sign, and verify the Relay product input. Use the registryctl CLI reference for the exact baseline and bundle commands.
Stage Relay without caller traffic. Require health, readiness, audit, and redacted deployment posture to pass.
Run one authorized, bounded source canary when the source authority permits that test. Offline preflight does not contact the source.
Move callers to the staged Relay and retire the old secret or trust material after the overlap window closes.

Private CA and mutual TLS changes stay in Relay. Do not copy source certificates, keys, or destinations into a Notary bundle.

Rotate caller keys and the Relay workload token

For a Relay API key, generate a new raw key and fingerprint:

registry-relay generate-api-key --id caller-rotation-2026-01

Store the emitted raw key only in the authorized client secret store. Store the fingerprint under the configured Relay secret reference, keep both caller entries during the overlap window, restart or roll Relay, move the caller, then remove the old entry. Relay does not provide live keyring reload in the documented v1 path.

For the Notary-to-Relay workload token, update the Relay workload policy and atomically replace the owner-readable token file used by Notary. Stage Relay first, then stage Notary against Relay and require the complete consultation contract check. Admit caller traffic only after both products are ready.

Rotate configuration signers and trust anchors

Create or inspect product-specific trust-anchor operations with:
Terminal window
```
registryctl anchor --help
```
Add the replacement public signer to the affected product’s local trust anchor before a bundle depends on that signer.
Distribute the updated anchor through the deployment trust path and restart staged instances.
Sign the next product bundle with the replacement key and a strictly higher sequence.
Run the stateless bundle check against the affected product anchor:
Terminal window
```
SIGNED_PRODUCT_BUNDLE=operator-inputs/signed-relay-bundle
PRODUCT_TRUST_ANCHOR=operator-inputs/relay-trust-anchor.json
registryctl bundle verify \
  --bundle-dir "$SIGNED_PRODUCT_BUNDLE" \
  --anchor-path "$PRODUCT_TRUST_ANCHOR"
```
This command verifies the signature, product binding, trust anchor, and signed file closure. It does not read product anti-rollback state or establish local rollback eligibility.

For an existing Relay state lineage, validate the later candidate or restart read-only against that node’s accepted state:

SIGNED_RELAY_BUNDLE=operator-inputs/signed-relay-bundle
RELAY_TRUST_ANCHOR=operator-inputs/relay-trust-anchor.json
RELAY_ROLLBACK_STATE=operator-state/relay-config-state.json
registry-relay config verify-bundle \
  --bundle-dir "$SIGNED_RELAY_BUNDLE" \
  --anchor-path "$RELAY_TRUST_ANCHOR" \
  --state-path "$RELAY_ROLLBACK_STATE"

For an existing Notary state lineage, run the equivalent Notary-owned check:
Terminal window
```
SIGNED_NOTARY_BUNDLE=operator-inputs/signed-notary-bundle
NOTARY_TRUST_ANCHOR=operator-inputs/notary-trust-anchor.json
NOTARY_ROLLBACK_STATE=operator-state/notary-config-state.json
registry-notary config verify-bundle \
  --bundle-dir "$SIGNED_NOTARY_BUNDLE" \
  --anchor-path "$NOTARY_TRUST_ANCHOR" \
  --state-path "$NOTARY_ROLLBACK_STATE"
```
Each product command verifies the candidate against that product’s existing local state without persisting an acceptance. These read-only commands require accepted state to exist and are for later candidate validation and restarts. A genuinely absent, version-specific state path is initialized only by the documented product boot path, not by either read-only verifier.
Remove or disable the old signer only after every retained deployment and rollback set has an accepted recovery path.

Repeat this workflow independently for Relay and Notary when both signers change. Do not interpret two successful verifications as atomic project activation.

Rotate Notary evidence-signing keys

Add a new key with a new kid and status: active. Move the intended credential or federation profiles to the new named key. Change the old key to publish_only for the full verifier and credential lifetime window, then disable or remove the old key through its governed cleanup path. Never reuse a kid for different key material.

The Notary signing key provider guide defines local JWK and PKCS#11 provider configuration, readiness behavior, and the complete key lifecycle.

Audit hash-secret rotation is a separate evidence-lifecycle event. Retain the old secret under the audit retention policy because changing the secret breaks lookup correlation with older pseudonyms.

Expected evidence

Retain:

The redacted preflight and semantic-comparison reports.
The verified product bundle report and product-specific sequence.
Health, readiness, audit-write, and redacted posture results from the staged instance.
A synthetic or authorized bounded canary result.
The old-material retirement time and the approved overlap window.

Evidence records may contain key ids, product ids, scopes, and certificate metadata. They must not contain raw keys, fingerprints, private certificates, tokens, environment values, or full configuration dumps.

What this proves

The offline gates prove that required references exist, file posture is acceptable, the environment does not widen the authored contract, and the generated product input validates. registryctl bundle verify proves the stateless signature, binding, trust-anchor, and file-closure checks. The product-owned config verify-bundle command additionally proves read-only local rollback eligibility against that product’s existing accepted state. Neither command persists bundle acceptance. Staged readiness and a bounded canary prove the tested runtime path.

These gates do not prove country approval, legal authority, every source operation, every caller migration, remote audit retention, or live interoperability that was not tested.

Roll back or recover

Before the replacement serves traffic, restore the prior secret reference, anchor, bundle, and runtime recovery set. After a higher configuration sequence has been accepted, do not delete anti-rollback state or activate an older sequence. Issue a reviewed higher-sequence recovery bundle, or use the exact documented, expiring local break-glass process.

Keep old Notary public verification keys published while credentials signed by those keys can still be verified. Keep both products staged and traffic blocked when Relay and Notary no longer agree on the consultation contract.

Escalate

Escalate to the product security owner when material may be exposed, a private key or token reaches logs, trust cannot overlap safely, a certificate changes source identity, an old Notary public key cannot remain published, anti-rollback state is missing, or a canary requires live country data.