Registry stack documentation: machine-readable Markdown.
Index of all pages: https://docs.registrystack.org/llms.txt
Full corpus: https://docs.registrystack.org/llms-full.txt

# Upgrade and roll back a deployment

> How to move a self-hosted Registry Relay or Registry Notary deployment to a new release, verify it, and revert if needed.

Use this procedure to move a self-hosted Registry Relay or Registry Notary deployment from
one release to another, and to revert to the previous release if the new one misbehaves.
It applies to deployments that consume release artifacts (binaries or the
`ghcr.io/registrystack/*` container images) pinned to a version tag or image digest.

One term needs disambiguating up front. This page uses "roll back" for redeploying the
previous release of the software. The governed-configuration feature has a separate control
called anti-rollback, which rejects config bundles older than the last applied sequence.
The two interact, but anti-rollback state formats can also change between releases. Keep
version-specific config and state restore sets, and restore the previous release's exact
anti-rollback state before starting its binary. The
[revert section](#roll-back-to-the-previous-release) covers this.

This page remains draft until the complete procedure passes against the exact independently
verified 1.0 candidate and a pinned standalone Solmara release.
The candidate-neutral record template is
`release/exercises/upgrade-exercise-v1.template.json`; a validated template is preparation, not
release evidence.

## Prerequisites

- Your deployment pins a version tag (`v0.13.0`) or an image digest, never `latest`.
- You know where your deployment keeps its persistent state. At minimum: the audit log
  directory of each product, the anti-rollback state file if `config_trust` is enabled
  (`antirollback_state_path`), each PostgreSQL database backing Relay consultation or Registry
  Notary correctness state, the matching role-provisioning and key-lifecycle records, and your
  config files and secrets. For generated and single-node
  Compose deployments, follow [Back up and restore a deployment](../backup-and-restore/)
  before changing the running containers. Store each release's config and anti-rollback
  state as a separate restore set; do not overwrite an older set with a newer file.
- You can run the release-verification steps in
  [SECURITY.md](https://github.com/registrystack/registry-stack/blob/v0.13.0/SECURITY.md)
  (cosign, and slsa-verifier for releases with provenance).

## Upgrade

1. Read the release notes and CHANGELOGs for every release between your current version and
   the target, in order. Collect every `BREAKING:` and `Deprecated` entry that names a config
   key, route, metric, or environment variable you use. Within a major line there are no
   breaking changes once the stack is at `v1.0.0`; before that, minors may contain them.
   Follow every sequential release hop unless the target release explicitly certifies a direct
   state path.

2. Download and verify the target release artifacts (cosign signature and, where present,
   SLSA provenance) before anything else touches them.

   When upgrading `registryctl` to `v0.9.0` or later, keep its versioned
   `registryctl-vX.Y.Z-image-lock.json` beside the binary and verify both against the
   release `SHA256SUMS` and available authenticity evidence. Existing generated projects
   can keep starting from the immutable image references already stored in their project
   files. Any later `registryctl init` or `registryctl add` requires the matching new lock
   before it changes those files. Do not reuse a lock from another CLI version or replace
   its digest references with mutable tags. For an intentionally separate verified lock
   location, set `REGISTRYCTL_IMAGE_LOCK` to that exact file.

3. Validate your current config against the committed Draft 2020-12 product schemas at
   `schemas/registry-relay.config.schema.json` and
   `schemas/registry-notary.config.schema.json`.
   Then validate it against the new binary offline, before deploying it:

   ```sh
   registry-relay doctor --config <your-config.yaml> --format json
   registry-notary doctor --config <your-config.yaml> --format json
   ```

   Run the new binary's `doctor`, not the old one's. It applies the new release's validation,
   including rejection of config keys the new release removed, and evaluates the deployment
   profile gates. Resolve every startup-failure finding before proceeding. Use
   `explain-config` to confirm the resolved configuration and required environment variables
   match your expectation.

4. Back up state that a failed upgrade must not lose: `secrets/local.env`, the audit log
   directory (copy, never truncate or edit; the hash chain makes hand-edited files
   unverifiable), the anti-rollback state file, every complete Relay consultation and Notary
   PostgreSQL database, and your config and source files as currently deployed. Record hashes and
   permissions without printing secrets, label the restore set with the running release, and
   leave it unchanged when the target release creates or updates its own state.

5. For a consultation-enabled Relay upgrade, remove Notary caller traffic, stop the active Relay
   fence holder, and prevent another Relay process from acquiring the fence. Run the target
   release's `registry-relay consultation bootstrap-state` with the existing database and the
   exact recorded owner, role connections, lock keys, chain-key epoch, active key, deadline, and
   retention inputs. The command is a clean-or-attested exact-schema installer, not a general
   migration framework. Continue only when it reports `installed_or_attested` and `identical`.
   If it reports capability or keyring drift, abort the upgrade without dropping schemas or
   changing bootstrap inputs. Start one target Relay without traffic and require `/ready` to
   return `200` before admitting callers.

6. For a Notary schema upgrade, remove traffic and stop every old Notary writer. Run the target
   release's `state install` as the migration role, then run `state doctor` as the runtime role.
   Do not run old and new Notary binaries concurrently across a schema change. Start one target
   replica without traffic and admit the rest only after every replica attests the same schema
   fingerprint. Keep the old release's binary or image reference, config, database backup, role
   provisioning, and sensitive-state key version until the soak period ends.
   Never start the previous Notary binary against the target schema, including as a diagnostic
   shortcut.

7. Verify the new deployment:

   - `/healthz` returns 200 and `/ready` returns 200 on both products.
   - On Registry Relay, a 503 from `/ready` with a `deployment/not_ready` problem body means a
     profile gate is failing; check the admin posture surface for the finding.
   - On Registry Notary, a 503 from `/ready` returns the generic `readiness.not_ready` problem
     body for any failing readiness check, including a profile gate; check the admin posture
     surface's deployment object to confirm whether a gate finding caused the failure.
   - The audit sink is writing: confirm new records appear in the audit log after a probe
     request.
   - A canary request per critical path: one governed read and, when enabled, one native
     consultation on Registry Relay; one claim evaluation and, if you issue credentials, one
     issuance on Registry Notary.
   - Metrics are flowing to your monitoring.

   This procedure has been partially exercised against a real topology: the
   [v0.8.3 -> v0.8.4 exercise record](https://github.com/registrystack/registry-stack/blob/v0.13.0/release/exercises/upgrade-v0.8.3-to-v0.8.4.md)
   covers the upgrade and roll-back steps above; release-artifact verification, credential
   issuance, metrics, the historical Redis replay/nonce path used by that release, and
   anti-rollback monotonicity were not exercised by that run. Current Registry Notary releases
   keep replay and nonce correctness state in PostgreSQL.

   The current
   [v0.10.0 -> v0.11.0 -> v0.10.0 release-gate procedure](https://github.com/registrystack/registry-stack/blob/v0.13.0/release/exercises/upgrade-v0.10.0-to-v0.11.0.md)
   adds old-evaluation rejection and re-evaluation before issuance,
   candidate-neutral multiple-credential rejection, and retained audit-chain
   recovery. It is a prepared procedure and does not claim a successful run
   until finalized-source and published-asset evidence are attached.

8. Keep the previous release's artifacts, config, exact anti-rollback state, database backups,
   role records, and key-lifecycle restore sets until the new release has run through your soak
   period. Only then remove them under your retention policy.

## Roll back to the previous release

Before target-version traffic is admitted, the general rollback path restores the complete
previous-release recovery set.
After the target accepts traffic that writes correctness state, audit evidence, or credentials,
fix forward by default.
Rollback after target writes is permitted only when that release documents and proves a safe
recovery conversion.

If rollback remains permitted:

1. Stop the target release. Restore the previous release's binary or image, the config that
   was running before the upgrade, and that release's exact anti-rollback state at its
   configured path. For Notary, also restore the matching complete database backup, role
   provisioning, and sensitive-state key version. Retain the target release's state separately.
   Never start the previous binary against a newer schema or anti-rollback state file.
   For a consultation-enabled Relay, reuse the current database only when the previous release's
   bootstrap command attests the same schema and keyring. Restore a pre-upgrade Relay database
   only when the target release acknowledged no consultation after that recovery point. If target
   traffic wrote state, keep the current database and fix forward unless a release-specific
   recovery procedure proves a safe conversion.

2. Respect the constraints that survive a software roll-back:

   - Anti-rollback state is release-specific durable evidence. Restore the exact prior
     release file, including its hash and permissions. Do not delete or reinitialize either
     release's state to make a bundle boot, and do not copy a newer state file over the
     prior restore set. Sequence monotonicity continues within each retained state lineage.
   - The audit chain must stay continuous. Never truncate or rewrite audit files to "match"
     the previous state; the chain verifies across the retained files, and the record of the
     failed upgrade is evidence, not noise.
   - Relay readiness does not prove restore freshness. A stale Relay database can omit durable
     consultation audit, replay, quota, materialization, or keyring history while retaining a
     valid catalog. Keep a questionable recovery point offline and recover forward to the last
     acknowledged write and external audit watermark.
   - If the upgrade rotated or introduced signing keys and credentials were issued with
     them, keep the new public keys published until those credentials expire; removing them
     breaks verification for holders. This mirrors the rollback plan in the Registry Notary
     deployment hardening runbook.
   - Do not rotate the audit hash secret as part of a roll-back; rotating it breaks
     correlation of records written before the rotation.

3. Re-run the verification list from the upgrade procedure against the reverted deployment.

4. Before retrying the upgrade, capture what failed: the problem responses, the posture
   findings, and the audit records from the failed window.

## Troubleshooting

- The new binary refuses to start with a config validation error naming a key you use: the
  key was removed or renamed. The error message and the release's migration notes name the
  replacement. Fix the config; do not downgrade to avoid the message.
- `/ready` stays 503 after upgrade: read the problem body. On Registry Relay,
  `deployment/not_ready` is a profile gate, inspect the admin posture surface;
  `schema/resource_unavailable` means a configured resource failed ingest. On Registry Notary,
  the problem body reports the generic code `readiness.not_ready` for any failing readiness
  check, including a profile gate; check the admin posture surface's deployment object to
  confirm whether a gate finding is the cause.
- Config bundle activation is rejected with a rollback outcome after reverting: confirm the
  previous release's exact config and anti-rollback state restore set is in place. Do not
  delete or reinitialize state. If the restored lineage itself has already accepted a newer
  sequence, issue a valid bundle with a higher sequence instead of activating an older one.
- Relay consultation bootstrap reports capability or keyring drift: the target binary, schema,
  roles, lock identity, or keyring lifecycle does not match the current database. Keep traffic
  blocked and restore the matching complete recovery set. Do not drop or rebootstrap the schemas.

## Next

- [Run single-node Compose behind your reverse proxy](../single-node-compose-behind-proxy/)
- [API stability and versioning](../../reference/api-stability/)
- [Back up and restore a deployment](../backup-and-restore/)
- [Security support window](../../security/support-window/)
- [Report a vulnerability](../../security/report-a-vulnerability/)