Registry stack documentation: machine-readable Markdown.
Index of all pages: https://docs.registrystack.org/v/0.38.0/llms.txt
Full corpus: https://docs.registrystack.org/v/0.38.0/llms-full.txt

# Upgrade and retire a deployment

> Upgrade Base Registry Engine, Registry Casework, Evidence, Registry Scheduling, and their clients to a new release in a safe order, and retire a deployment without losing the records it must leave behind.

Use this runbook when you move a deployment of Base Registry Engine (BReg), Registry Casework,
Evidence, and Registry Scheduling to a new release, or take it out of service. Each product's own guide carries its upgrade
command; this page gives the order across products, the version rule that order enforces, and
what each product must leave behind when it is retired.

## Run one release everywhere

Every Registry Stack artifact of a release shares one version: the runtimes, their adopter tooling,
and the Rust, Node.js, and Python clients. Run the same release on every side. There is no
supported version-skew window, between two runtimes or between a client and a runtime, and a
rolling upgrade that mixes releases is not supported:

- **Casework and BReg are checked.** Casework compares BReg's `Registry-Engine-Version` header with
  its own release on every contract read and refuses any other release; see
  [Upgrade Casework and BReg in lock-step](../../casework/#upgrade-casework-and-breg-in-lock-step).
- **Adopter tooling is checked.** `evidencectl` refuses an `evidence` or `bregctl` binary of
  another version.
- **Clients are not checked, and break silently.** Clients send no version, and runtimes accept any
  caller. The clients decode responses strictly, so a member one release adds can fail an older
  client's decoding, and a member one release requires can be missing from an older runtime's
  answer. Upgrade each client to the same release as the runtime it calls.
- **BReg activation is not rolling.** Once `bregctl apply` activates a successor, every `breg`
  process still running the previous package fails readiness until it restarts onto the successor.
  Plan the restart as part of the activation, not after it.

A release promises a forward state path from its immediate predecessor only, starting at
`v0.33.0`; skip no release without that release's notes certifying the direct path, and expect no
reverse path. See [Compatibility direction](../../../reference/api-stability/#compatibility-direction).

{/* Evidence: Cargo.toml, [workspace.package] version (every crate uses version.workspace);
    crates/registry-casework-breg/src/lib.rs, same_release() and PeerVersionMismatch;
    crates/registry-evidencectl/src/evidence_binary.rs and source_add.rs, check_public_bregctl();
    crates/registry-breg-client/src/response.rs, deny_unknown_fields;
    docs/site/src/content/docs/operate/breg-changes.mdx, "Activate the successor";
    release/scripts/rehearse-upgrade.py, FORWARD_PATH_FLOOR. */}

## Upgrade in this order

BReg goes first, because Casework and Evidence both read it. Casework follows because it refuses
a BReg of another release. Evidence and its wallet-facing front end come next. Scheduling reads
none of them and upgrades on its own, before the clients. Each release's notes list, per product,
the project and runtime-file changes its upgrade needs; make them in the step that upgrades that
product. Rehearse the whole
sequence on a restored copy first, as
[Run a disaster-recovery drill](../back-up-and-restore/#run-a-disaster-recovery-drill) describes.

1. **Back up every database.** Take the BReg and Casework `pg_dump`s, and the Scheduling one
   when it runs, and copy those products' packages and runtime files. These backups are the only way back; see
   [Roll back](#roll-back).
2. **Stop `evidence-oid4vci`**, when it runs. Its outstanding offers end here; see
   [Operational limits](../../../configure/evidence-oid4vci/#operational-limits).
3. **Pause and drain Evidence traffic** for every question that reads BReg, or pause the whole
   instance when you cannot drain one question at a time. Keep the previous Evidence candidate.
4. **Upgrade BReg.** Make the project and runtime-file changes the release notes list, rebuild
   the deployed project with no model change using the new `bregctl test` and `bregctl package
   --test-receipt FILE --output BUILD` (a receipt from the earlier release is not accepted), point
   `package.root` at `BUILD/package` (and `package.expectedDigest`, when set, at the package
   digest it reports, `packageDigest` with `--format json`), check it against the live database with
   `bregctl plan --runtime-config FILE --package BUILD/package` naming the same directory, apply
   it with `bregctl apply --runtime-config FILE --package BUILD/package`, run `bregctl verify --runtime-config
   FILE`, and restart every `breg` process on the new binary; `bregctl status --runtime-config
   FILE` confirms what the database activated. A model change follows as its own successor; see
   [Activate the successor](../../breg-changes/#activate-the-successor). From a release before
   the activation ledger, this first `apply` adopts the database as it stands: `package.root`
   and `--package` must both name the rebuilt package, which must describe the schema the
   database runs, `plan` reports `activation` `adopted`, the ledger history the earlier release
   kept is dropped, and the instance claim is carried over, or recorded when the database had
   none. Every ingestion run the earlier release opened against the running package is rebound
   to the adopted package and stays writable. Adoption records its own activation id on every
   import authority a pre-ledger release had already closed, because the revision each was
   opened under has no activation in the ledger, so those authorities' activation id names the
   adoption rather than the activation they were opened under. A `package.root` that still names
   the earlier release's package is refused with path `package.root`. When an `apply` stops
   before it finishes, rerun it with the same `database.roles`: a retry under other roles is
   refused as `apply.resume.roles_differ`, naming the roles the activation started with. A new package applied under `database.roles` other than the ones the active activation serves with is refused as `apply.successor.roles_differ` before maintenance; apply the active package under the new roles first, then the new package. A role change, the active package applied under other `database.roles`, cannot be assessed by `bregctl migration reconcile`: when one stops before it finishes, fix the cause and rerun the same apply, which resumes it. Casework reports each BReg source as unavailable from here until step 5
   finishes.
5. **Upgrade Casework.** Settle pending attempts first, since an upgrade can strand them. From a
   release before the activation ledger, add `identity.databaseId` to the runtime file. When step
   4 changed the `registryRevision` a BReg source serves, check each such source with `caseworkctl
   check PROJECT --against-breg-package DIR --source-id ID`, repin it with `caseworkctl source add
   BREG_PROJECT --project PROJECT --source-id ID --apply`, package the Casework project again, and
   point the runtime file's `package.root` at the new package (and `package.expectedDigest`, when
   set, at its digest); see
   [Check a BReg source's pinned revision](../../casework/#check-a-breg-sources-pinned-revision).
   Stop every earlier `casework` process before the apply: one left running holds the audit
   writer lock the new runtime needs, and can strand a source attempt when the apply changes a
   source's binding generation. Run `caseworkctl plan --runtime-config FILE` and `caseworkctl apply --runtime-config FILE` with the
   new binaries, then start `casework` and run `caseworkctl doctor --runtime-config FILE`; see [Plan, apply, and serve](../../casework/#plan-apply-and-serve).
6. **Upgrade Evidence.** Re-import each BReg source whose export the release notes say changed,
   then package a fresh candidate with the new `evidencectl package`, run `evidence check
   --runtime-config FILE --require-runtime-dependencies` with the new binary (add
   `--without-audit-lock` while the earlier instance still runs), stop the earlier `evidence`,
   start the new one, and verify a fresh synthetic assertion. Then start the new `evidence-oid4vci`.
7. **Upgrade Scheduling**, when it runs. Stop the earlier `scheduling` runtime, or keep its
   destination bindings until its hook deliveries drain. Add `identity.databaseId` to the runtime
   file when it has none, run `schedulingctl plan --runtime-config FILE` and `schedulingctl apply --runtime-config
   FILE` with the new binary, then start `scheduling`; `schedulingctl status --runtime-config
   FILE` confirms what the database activated. From a release before the activation ledger, the
   first `apply` adopts the database and backfills the retained policy document, and the runtime
   refuses to start until it has.
8. **Resume traffic, and upgrade clients.** Roll out every application that embeds a Registry
   Stack client at the same release before it calls the upgraded runtimes.

{/* Evidence: docs/site/src/content/docs/operate/casework.mdx, "Upgrade Casework and BReg in
    lock-step" (BReg first, readiness fails in between);
    docs/site/src/content/docs/configure/evidence-oid4vci.mdx, "Operational limits" (stop the
    adapter, restart Evidence, start the adapter);
    docs/site/src/content/docs/tutorials/deploy-evidence-from-breg.mdx, "Coordinate a breaking
    update"; crates/registry-casework/src/store.rs, StoreError::AttemptPending;
    crates/registry-breg/src/postgres/interlock.rs, adopt_pre_ledger_database() (ingestion run
    rebinding, import authority activation ids); crates/registry-breg/src/migration.rs,
    MigrationError::ResumeRolesDiffer;
    crates/registry-bregctl/src/apply_lifecycle.rs, load_active_predecessor_package();
    crates/registry-bregctl/src/lib.rs, PlanSuccessReport, StatusSuccessReport, and
    lifecycle_failure() for ApplyLifecycleError::CurrentPackage;
    release/notes/v0.36.0.md, "Upgrade Scheduling, Relay, and Discovery from v0.35.0"
    (identity.databaseId, apply before start, hook drain);
    release/scripts/rehearse-upgrade.py rehearses each product alone, not this composition. */}

## Roll back

No product migrates backwards. A Casework binary refuses a database whose schema is newer than it
supports, and a BReg package applies forward only. Evidence refuses a runtime file carrying keys
its release does not know. To return to the previous release:

- **BReg.** To return to the previous release, stop every `breg` and retire the upgraded
  database, restore the step 1 `pg_dump` into a fresh database, and point the previous runtime file copied
  in step 1 at it. The restored copy carries the claim of the database it was dumped from, so
  adopt it with the previous release's `bregctl instance-claim adopt --runtime-config FILE
  --acknowledge-original-retired` (see
  [After a logical restore](../back-up-and-restore/#after-a-logical-restore) for why), then
  start the previous `breg` with the previous package and that runtime file; the new release's
  tools cannot read the earlier package or runtime file. To undo only a model
  change, roll forward to a successor that reverts it, and when the data itself must go back,
  restore the pre-activation backup; see
  [Roll back by rolling forward](../../breg-changes/#roll-back-by-rolling-forward).
- **Casework.** Restore the pre-apply backup and run the previous binary, package, and runtime
  file copied in step 1; see
  [Restore Registry Casework](../back-up-and-restore/#restore-registry-casework) for what that
  restore loses.
- **Evidence.** Restart the previous binary with the previous bundle and runtime file.
- **Scheduling.** Restore the pre-apply database backup, then run the previous binary, package,
  and runtime file copied in step 1. Always restore first: an earlier Scheduling runtime may not
  detect a schema newer than its own.

Because the release rule holds in both directions, rolling back one product means rolling back
every product that must match it: Casework refuses a BReg that went back without it.

{/* Evidence: crates/registry-casework/src/store.rs, refuse_newer_schema();
    crates/registry-bregctl/src/lib.rs, LegacyFormat refused as apply.package.refused;
    crates/registry-bregctl/src/package_lifecycle.rs, PACKAGE_DIRECTORY (the package child of
    --output); crates/registry-breg/src/instance_claim.rs, module docs (a logical copy keeps its
    original's claim; v0.35.0 ships instance-claim adopt);
    crates/registry-platform-audit/src/writer.rs, single-writer lock;
    crates/registry-evidence/src/cli.rs, Check (--runtime-config, --without-audit-lock);
    crates/registry-breg/src/migration.rs, MigrationError::PackageBinding;
    crates/registry-evidence/src/config.rs, deny_unknown_fields on the runtime file. */}

## Retire a deployment

Retire the products in the reverse of the upgrade order, so nothing still running depends on what
you have removed. Retiring a product leaves records behind: signatures others still verify,
decisions others may challenge, and audit entries your retention policy holds. None of the products
has a decommission command, so the steps below are yours.

### Retire Scheduling

Scheduling reads no other product and no other product reads it, so it can retire at any point
in this sequence.

1. Stop routing booking requests to Scheduling, and tell the callers holding appointments how
   those appointments will be honoured.
2. Stop `scheduling`. Reminder and observer-hook deliveries are outbox work its own workers send,
   so every delivery still pending stops with it and is not sent.
3. Scheduling has no export command. Keep a final `pg_dump` for as long as your policy holds its
   appointment records.
4. Ship the final audit file, its sealed files, and the `schedulingctl` sibling of `audit.path`.

### Retire Evidence

1. Stop `evidence-oid4vci`, then stop routing requests to Evidence.
2. Keep the published public keys available. An assertion Evidence issued stays valid for up to
   `maximumAssertionValiditySeconds` (at most one year), and the JWKS is served only by the running
   process, so publish the key set elsewhere or give it to each relying party, and keep it until
   that period and the verifier clock skew have passed since the last issuance.
3. Ship the final audit files after the process stops, and keep every audit master whose
   pseudonyms an investigation may still need; see
   [Rotate the audit master](../../evidence-audit/#rotate-the-audit-master).
4. Retire the signing key in Transit only after step 2's period ends; see
   [Retire the old version](../../../tutorials/rotate-evidence-signing-keys/#retire-the-old-version).
5. Retire or rebind the Evidence provider in every BReg project whose governed actions call it,
   since an action that resolves against a retired provider fails.

### Retire Casework

1. Stop staff access. Settle every pending and uncertain attempt, as
   [Settle an uncertain source attempt](../../casework-retention/#settle-an-uncertain-source-attempt)
   describes, so no change is left half-applied at a source.
2. Finish or close the reviews Casework hosts for BReg. A review left open makes BReg wait for a
   result that never comes; close it from BReg with
   [`bregctl review-recovery close`](../../breg/#resubmit-or-close-a-review-the-authority-lost).
3. Casework has no export command. Keep a final `pg_dump` for as long as your policy holds its
   decisions and accountability records, or erase source-backed payload copies first with
   [`caseworkctl retention erase`](../../casework-retention/#erase-source-backed-payload-copies).
4. Stop the runtime, ship the final audit file, its sealed files, and the `caseworkctl` companion
   file, then revoke Casework's client credentials at each BReg source's identity provider.

### Retire Base Registry Engine

1. Stop every consumer, then close the import authorities still open with
   `bregctl import-authority close`.
2. Export what must outlive the registry. `bregctl data export` exports one entity through one
   access profile, checkpointed; a final `pg_dump` is the only complete copy. See
   [Export with a checkpoint](../../breg-data/#export-with-a-checkpoint).
3. Erase what your policy does not let you keep, before that final backup: see
   [Erase retained history](../../breg-retention/#erase-retained-history) and
   [Erase expired Evidence uses](../../breg-retention/#erase-expired-evidence-uses).
4. Stop `breg`. Pending webhook deliveries stop with it and are not sent; check
   `bregctl webhook list` first if receivers must see them.
5. Ship the final audit files and the `bregctl` companion file.
6. Keep the field-encryption key for as long as you keep any backup that holds sealed values, and
   the audit hash key for as long as you keep the audit archive. Destroy each only when the last
   record that needs it is gone.

{/* Evidence: crates/registry-evidence/src/config.rs, maximumAssertionValiditySeconds bounds;
    docs/site/src/content/docs/reference/api-stability.mdx, JWKS served by the process;
    products/evidence/OPERATOR-CONTRACT.md, predecessor key retention and audit master retention;
    crates/registry-caseworkctl/src/lib.rs, no export command, RetentionCommand, AttemptCommand;
    crates/registry-schedulingctl/src/lib.rs, no export command; crates/registry-scheduling/src/runtime.rs,
    outbox workers spawned by the runtime; products/scheduling/README.md, the schedulingctl audit file;
    crates/registry-bregctl/src/lib.rs, DataCommand::Export, ImportAuthorityCommand,
    WebhookCommand; crates/registry-breg/src/field_encryption.rs, module docs. */}

## Next

- [Back up, restore, and drill recovery](../back-up-and-restore/)
- [Change an active registry](../../breg-changes/)
- [Security support window](../../../security/support-window/)
- [API stability and versioning](../../../reference/api-stability/)