Upgrade and roll back a deployment

View as Markdown

Use this procedure to move a self-hosted Registry Relay or Registry Notary deployment from one release to another, and to revert to the previous release if the new one misbehaves. It applies to deployments that consume release artifacts (binaries or the ghcr.io/registrystack/* container images) pinned to a version tag or image digest.

One term needs disambiguating up front. This page uses “roll back” for redeploying the previous release of the software. The governed-configuration feature has a separate control called anti-rollback, which rejects config bundles older than the last applied sequence. The two interact, but anti-rollback state formats can also change between releases. Keep version-specific config and state restore sets, and restore the previous release’s exact anti-rollback state before starting its binary. The revert section covers this.

This page remains draft until the complete procedure passes against the exact independently verified 1.0 candidate and a pinned standalone Solmara release. The candidate-neutral record template is release/exercises/upgrade-exercise-v1.template.json; a validated template is preparation, not release evidence.

Prerequisites

Your deployment pins a version tag (v0.13.0) or an image digest, never latest.
You know where your deployment keeps its persistent state. At minimum: the audit log directory of each product, the anti-rollback state file if config_trust is enabled (antirollback_state_path), each PostgreSQL database backing Relay consultation or Registry Notary correctness state, the matching role-provisioning and key-lifecycle records, and your config files and secrets. For generated and single-node Compose deployments, follow Back up and restore a deployment before changing the running containers. Store each release’s config and anti-rollback state as a separate restore set; do not overwrite an older set with a newer file.
You can run the release-verification steps in SECURITY.md (cosign, and slsa-verifier for releases with provenance).

Upgrade

Read the release notes and CHANGELOGs for every release between your current version and the target, in order. Collect every BREAKING: and Deprecated entry that names a config key, route, metric, or environment variable you use. Within a major line there are no breaking changes once the stack is at v1.0.0; before that, minors may contain them. Follow every sequential release hop unless the target release explicitly certifies a direct state path.
Download and verify the target release artifacts (cosign signature and, where present, SLSA provenance) before anything else touches them.

When upgrading registryctl to v0.9.0 or later, keep its versioned registryctl-vX.Y.Z-image-lock.json beside the binary and verify both against the release SHA256SUMS and available authenticity evidence. Existing generated projects can keep starting from the immutable image references already stored in their project files. Any later registryctl init or registryctl add requires the matching new lock before it changes those files. Do not reuse a lock from another CLI version or replace its digest references with mutable tags. For an intentionally separate verified lock location, set REGISTRYCTL_IMAGE_LOCK to that exact file.
Validate your current config against the committed Draft 2020-12 product schemas at schemas/registry-relay.config.schema.json and schemas/registry-notary.config.schema.json. Then validate it against the new binary offline, before deploying it:
Terminal window
```
registry-relay doctor --config <your-config.yaml> --format json
registry-notary doctor --config <your-config.yaml> --format json
```
Run the new binary’s doctor, not the old one’s. It applies the new release’s validation, including rejection of config keys the new release removed, and evaluates the deployment profile gates. Resolve every startup-failure finding before proceeding. Use explain-config to confirm the resolved configuration and required environment variables match your expectation.
Back up state that a failed upgrade must not lose: secrets/local.env, the audit log directory (copy, never truncate or edit; the hash chain makes hand-edited files unverifiable), the anti-rollback state file, every complete Relay consultation and Notary PostgreSQL database, and your config and source files as currently deployed. Record hashes and permissions without printing secrets, label the restore set with the running release, and leave it unchanged when the target release creates or updates its own state.
For a consultation-enabled Relay upgrade, remove Notary caller traffic, stop the active Relay fence holder, and prevent another Relay process from acquiring the fence. Run the target release’s registry-relay consultation bootstrap-state with the existing database and the exact recorded owner, role connections, lock keys, chain-key epoch, active key, deadline, and retention inputs. The command is a clean-or-attested exact-schema installer, not a general migration framework. Continue only when it reports installed_or_attested and identical. If it reports capability or keyring drift, abort the upgrade without dropping schemas or changing bootstrap inputs. Start one target Relay without traffic and require /ready to return 200 before admitting callers.
For a Notary schema upgrade, remove traffic and stop every old Notary writer. Run the target release’s state install as the migration role, then run state doctor as the runtime role. Do not run old and new Notary binaries concurrently across a schema change. Start one target replica without traffic and admit the rest only after every replica attests the same schema fingerprint. Keep the old release’s binary or image reference, config, database backup, role provisioning, and sensitive-state key version until the soak period ends. Never start the previous Notary binary against the target schema, including as a diagnostic shortcut.
Verify the new deployment:
- /healthz returns 200 and /ready returns 200 on both products.
- On Registry Relay, a 503 from /ready with a deployment/not_ready problem body means a profile gate is failing; check the admin posture surface for the finding.
- On Registry Notary, a 503 from /ready returns the generic readiness.not_ready problem body for any failing readiness check, including a profile gate; check the admin posture surface’s deployment object to confirm whether a gate finding caused the failure.
- The audit sink is writing: confirm new records appear in the audit log after a probe request.
- A canary request per critical path: one governed read and, when enabled, one native consultation on Registry Relay; one claim evaluation and, if you issue credentials, one issuance on Registry Notary.
- Metrics are flowing to your monitoring.
This procedure has been partially exercised against a real topology: the v0.8.3 -> v0.8.4 exercise record covers the upgrade and roll-back steps above; release-artifact verification, credential issuance, metrics, the historical Redis replay/nonce path used by that release, and anti-rollback monotonicity were not exercised by that run. Current Registry Notary releases keep replay and nonce correctness state in PostgreSQL.

The current v0.10.0 -> v0.11.0 -> v0.10.0 release-gate procedure adds old-evaluation rejection and re-evaluation before issuance, candidate-neutral multiple-credential rejection, and retained audit-chain recovery. It is a prepared procedure and does not claim a successful run until finalized-source and published-asset evidence are attached.
Keep the previous release’s artifacts, config, exact anti-rollback state, database backups, role records, and key-lifecycle restore sets until the new release has run through your soak period. Only then remove them under your retention policy.

Roll back to the previous release

Before target-version traffic is admitted, the general rollback path restores the complete previous-release recovery set. After the target accepts traffic that writes correctness state, audit evidence, or credentials, fix forward by default. Rollback after target writes is permitted only when that release documents and proves a safe recovery conversion.

If rollback remains permitted:

Stop the target release. Restore the previous release’s binary or image, the config that was running before the upgrade, and that release’s exact anti-rollback state at its configured path. For Notary, also restore the matching complete database backup, role provisioning, and sensitive-state key version. Retain the target release’s state separately. Never start the previous binary against a newer schema or anti-rollback state file. For a consultation-enabled Relay, reuse the current database only when the previous release’s bootstrap command attests the same schema and keyring. Restore a pre-upgrade Relay database only when the target release acknowledged no consultation after that recovery point. If target traffic wrote state, keep the current database and fix forward unless a release-specific recovery procedure proves a safe conversion.
Respect the constraints that survive a software roll-back:
- Anti-rollback state is release-specific durable evidence. Restore the exact prior release file, including its hash and permissions. Do not delete or reinitialize either release’s state to make a bundle boot, and do not copy a newer state file over the prior restore set. Sequence monotonicity continues within each retained state lineage.
- The audit chain must stay continuous. Never truncate or rewrite audit files to “match” the previous state; the chain verifies across the retained files, and the record of the failed upgrade is evidence, not noise.
- Relay readiness does not prove restore freshness. A stale Relay database can omit durable consultation audit, replay, quota, materialization, or keyring history while retaining a valid catalog. Keep a questionable recovery point offline and recover forward to the last acknowledged write and external audit watermark.
- If the upgrade rotated or introduced signing keys and credentials were issued with them, keep the new public keys published until those credentials expire; removing them breaks verification for holders. This mirrors the rollback plan in the Registry Notary deployment hardening runbook.
- Do not rotate the audit hash secret as part of a roll-back; rotating it breaks correlation of records written before the rotation.
Re-run the verification list from the upgrade procedure against the reverted deployment.
Before retrying the upgrade, capture what failed: the problem responses, the posture findings, and the audit records from the failed window.

Troubleshooting

The new binary refuses to start with a config validation error naming a key you use: the key was removed or renamed. The error message and the release’s migration notes name the replacement. Fix the config; do not downgrade to avoid the message.
/ready stays 503 after upgrade: read the problem body. On Registry Relay, deployment/not_ready is a profile gate, inspect the admin posture surface; schema/resource_unavailable means a configured resource failed ingest. On Registry Notary, the problem body reports the generic code readiness.not_ready for any failing readiness check, including a profile gate; check the admin posture surface’s deployment object to confirm whether a gate finding is the cause.
Config bundle activation is rejected with a rollback outcome after reverting: confirm the previous release’s exact config and anti-rollback state restore set is in place. Do not delete or reinitialize state. If the restored lineage itself has already accepted a newer sequence, issue a valid bundle with a higher sequence instead of activating an older one.
Relay consultation bootstrap reports capability or keyring drift: the target binary, schema, roles, lock identity, or keyring lifecycle does not match the current database. Keep traffic blocked and restore the matching complete recovery set. Do not drop or rebootstrap the schemas.