Registry stack documentation: machine-readable Markdown.
Index of all pages: https://docs.registrystack.org/preview/llms.txt
Full corpus: https://docs.registrystack.org/preview/llms-full.txt

# Back up and restore a deployment

> How to preserve generated secrets, Relay and Notary PostgreSQL state, and config-trust files across container recreation and upgrades.

Use this procedure before you recreate containers, move a host, or upgrade a
single-node Registry Relay or Registry Notary deployment. Start with `secrets/local.env`
because losing it rotates generated API credentials, audit HMAC secrets, and the local
Notary issuer identity; then preserve product state directories, every enabled product's
complete PostgreSQL database, config-trust files, and source inputs.

This page remains draft until complete backup and restore pass against the exact independently
verified 1.0 candidate and its pinned standalone Solmara topology.

## When to use this

Use this for `registryctl` generated projects and for hand-written single-node Docker Compose
deployments that keep state on the host running the containers.
Use your platform's snapshot, secret-manager, and database backup process for hosted,
multi-node, or Kubernetes deployments.

This procedure preserves durable state.
It does not preserve process-local state such as OpenID Connect (OIDC) JSON Web Key Set
(JWKS) caches or Relay source-integration token caches. A restart can still interrupt an
in-flight OpenID for Verifiable Credential Issuance browser journey even when its bounded
preauthorization state survives in PostgreSQL.

## Before you start

- Know the project directory that contains `compose.yaml`, `registry-stack.yaml`, `secrets/`,
  and `state/`.
- Know whether Relay consultations or Registry Notary are enabled, which PostgreSQL database each
  product uses, and which role-provisioning record belongs to each database.
- For a deployment with audited SnapshotExact materializations, remove dependent traffic, stop
  the Relay fence holder and every Notary caller, and prevent replacement processes from starting.
  Keep that topology quiesced while capturing source inputs, the Relay ingest cache, and the Relay
  consultation database at one coordinated recovery point.
- For a deployment without audited SnapshotExact, stop write traffic or use a filesystem snapshot
  if you need an exact point-in-time copy of audit files.
- Keep backups encrypted and access-controlled. `secrets/local.env`, audit files,
  config-trust state, PostgreSQL correctness state, and sensitive-state key versions are
  security-sensitive.

## Back up generated secrets first

Create a backup directory and copy `secrets/local.env` before copying other state:

```sh
backup_dir="backup/registry-stack-$(date +%Y%m%d)"
install -d -m 700 "$backup_dir"
cp -p secrets/local.env "$backup_dir/local.env"
sha256sum "$backup_dir/local.env" > "$backup_dir/local.env.sha256"
chmod 600 "$backup_dir/local.env" "$backup_dir/local.env.sha256"
```

Expected output is no output.
The checksum file lets you confirm that a later restore put back the exact same secret file.

## Back up config, source files, and product state

Back up the project manifest when present, Compose file, product config directories, source data,
and host-mounted state:

```sh
backup_items=""
for item in registry-stack.yaml compose.yaml relay notary state data; do
  [ -e "$item" ] && backup_items="$backup_items $item"
done
[ -f .env ] && backup_items="$backup_items .env"
tar -cf "$backup_dir/config-source-state.tar" $backup_items
sha256sum "$backup_dir/config-source-state.tar" > "$backup_dir/config-source-state.tar.sha256"
```

Expected output is one checksum line in `config-source-state.tar.sha256`.
The archive includes `state/relay/` and `state/notary/` when those directories exist.
Those paths are where generated Compose projects mount Relay cache, optional file-audit
directories, and config-trust anti-rollback state if you enable it with
`antirollback_state_path`.

When audited SnapshotExact is configured, run this archive command only after the topology is
quiesced. Keep Relay and its Notary callers stopped until the matching Relay database backup
finishes. A source archive, ingest-cache archive, and Relay database taken at independent live
times do not form a recoverable materialization set.

If your hand-written config uses different paths, include them in the archive:

- Relay or Notary file-audit directories, such as `/var/log/registry-relay/` or
  `/var/log/registry-notary/`.
- `config_trust.trust_anchor_path`, `config_trust.bundle_path`, and
  `config_trust.antirollback_state_path`.
- Source files, metadata manifests, signed bundles, and trust anchors that are mounted
  into the containers.

## Back up Relay consultation PostgreSQL state

Skip this section when Relay has no `consultation` configuration. The ordinary Relay ingest cache
remains covered by the file archive.

The consultation state plane is a complete Relay-owned correctness boundary. Its
`relay_state_private` and `relay_state_api` schemas contain durable consultation audit,
completion and replay decisions, quotas, dispatch fencing, materialization publication state,
and audit-pseudonym keyring metadata. They are separate from Notary's
`registry_notary_private` and `registry_notary_api` schemas. Never copy state tables between the
products.

Remove traffic from Notary callers first, then stop the Relay fence holder and prevent another
Relay process from starting. Keep the source inputs and Relay ingest cache unchanged while the
database backup is captured. If Relay and Notary form one service, keep both products quiesced
while their source, cache, and database backups are taken or use a platform snapshot that gives
all stores one coordinated recovery point. A transactionally consistent dump of one live database
does not make independently backed-up source, cache, or product databases share a recovery point.

For a database dedicated to Relay and an empty-database restore in the same PostgreSQL cluster,
where the pre-created roles keep their object identifiers, back up the complete Relay database
through the migration login:

```sh
PGSERVICEFILE=/run/registry-relay/pg_service.conf \
PGSERVICE=registry_relay_migration \
PGPASSFILE=/run/secrets/registry-relay-migration.pgpass \
pg_dump --format=custom --no-owner --no-acl \
  --role=relay_state_owner \
  --file="$backup_dir/relay-consultation-state.dump" \
  --dbname=registry_relay
sha256sum "$backup_dir/relay-consultation-state.dump" \
  > "$backup_dir/relay-consultation-state.dump.sha256"
```

If Relay shares a database with another owner, have the database administrator back up the
complete database with an appropriately privileged backup identity. Do not narrow the backup to
the two Relay schemas.

Expected output is one checksum line. Store the Relay release, config and signed artifact closure,
PostgreSQL major, all five role names, the four bound role object identifiers, bootstrap inputs,
keyring lifecycle settings, PostgreSQL trust root, and retained audit-pseudonym key material with
the encrypted recovery record. Keep secret values outside the database dump.

For every audited SnapshotExact binding, the access-controlled recovery record must bind:

- The source-input archive SHA-256 and the exact source revision.
- The Relay ingest-cache archive SHA-256.
- The complete Relay database backup SHA-256, including materialization active-pointer and history
  tables.
- The exact active binding, publication sequence, generation, and restricted content digest.
- The time or write-ahead-log position shared by the quiesced recovery set.

Hash and sign or independently attest that closed recovery record. Do not put restricted content
digests, source paths, rows, credentials, or key material in general logs or a public release
record. A list of unrelated artifact hashes is not evidence that the stores belong to one
generation.

For recovery into another PostgreSQL cluster, use a physical backup or cluster-level managed
snapshot that preserves the PostgreSQL role catalog and its object identifiers. Recreating the
same role names does not preserve those identifiers, and Relay intentionally rejects the restored
binding as capability drift. The
[Relay product documentation](../../products/registry-relay/) documents the schema and role
boundary in full.

## Back up PostgreSQL when Notary is present

Back up the complete Notary database at one consistent point. Do not back up individual
correctness tables. Stop Notary writers for a quiesced backup, or use the database platform's
transactionally consistent snapshot workflow. Use a PostgreSQL service definition and password
file so credentials do not appear in process arguments:

```ini
# /run/registry-notary/pg_service.conf
[registry_notary_migrator]
host=postgres.example.internal
port=5432
dbname=registry_notary
user=registry_notary_migrator
sslmode=verify-full
sslrootcert=/run/secrets/notary-postgres-ca.pem
```

The service name is operator-defined. The
[Registry Notary PostgreSQL operations guide](https://docs.registrystack.org/products/registry-notary/postgresql-state-operations/)
provides the complete role-provisioning example and password-file requirements.

```sh
PGSERVICEFILE=/run/registry-notary/pg_service.conf \
PGSERVICE=registry_notary_migrator \
PGPASSFILE=/run/secrets/registry-notary-migrator.pgpass \
pg_dump --format=custom --no-owner --no-acl \
  --role=registry_notary_owner \
  --file="$backup_dir/notary-state.dump" \
  --dbname=registry_notary
sha256sum "$backup_dir/notary-state.dump" \
  > "$backup_dir/notary-state.dump.sha256"
```

Expected output is one checksum line. Preserve the application release, migration set,
PostgreSQL major, reproducible owner and runtime role provisioning, recovery timestamp, and
sensitive-state key version with the encrypted backup. Keep the key value in the secret manager,
outside the database dump.

## Restore files

On a new host, install Docker Compose and `registryctl`, create the project directory, and copy
the backup directory into it.
On the original host, stop the generated services before restoring files:

```sh
registryctl stop
```

Expected output comes from Docker Compose and shows the containers stopping or already stopped.
Then restore secrets and host-mounted state:

```sh
sha256sum -c "$backup_dir/local.env.sha256"
install -d -m 700 secrets
cp -p "$backup_dir/local.env" secrets/local.env
sha256sum -c "$backup_dir/config-source-state.tar.sha256"
tar -xf "$backup_dir/config-source-state.tar"
```

Both checksum commands print `OK` for the named file.
Do not generate a new `secrets/local.env` to replace a lost one unless you intend to rotate
the deployment's API credentials, audit HMAC secret, and local Notary issuer key.
On a new host, compare `.env` with `id -u` and `id -g`.
If `REGISTRY_STACK_RUNTIME_UID` or `REGISTRY_STACK_RUNTIME_GID` names the old host owner, edit
those two values before starting containers so Relay and Notary can write the restored private
state directories.
If the archive was extracted by root or with numeric owners preserved, align restored state
ownership and private permissions with the runtime identity:

```sh
runtime_uid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_UID" { print $2 }' .env)"
runtime_gid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_GID" { print $2 }' .env)"
if [ -n "$runtime_uid" ] && [ -n "$runtime_gid" ] && [ -d state ]; then
  sudo chown -R "$runtime_uid:$runtime_gid" state
  find state -type d -exec chmod 700 {} +
  find state -type f -exec chmod 600 {} +
fi
```

For audited SnapshotExact, do not start Relay after restoring the files. First verify that the
source-input archive, ingest-cache archive, and Relay database dump match the same attested
recovery record. The record's active generation and restricted content digest must agree with the
database active pointer and history and with the retained cache generation.

## Restore Relay consultation PostgreSQL state

Skip this section when no Relay consultation dump or snapshot is present. Keep Relay and every
Notary caller offline while restoring into an empty, isolated, writable PostgreSQL 16 through 18
primary.

For a same-cluster logical restore, retain the original owner, runtime, keyring-maintenance, and
keyring-reader roles and their object identifiers. Restore the complete dump as the original
owner. For a different cluster, restore the physical backup that includes the original role
catalog. A logical restore into newly recreated role names is not a supported rebinding path.

```sh
sha256sum -c "$backup_dir/relay-consultation-state.dump.sha256"
PGSERVICEFILE=/run/registry-relay/pg_service.conf \
PGSERVICE=registry_relay_migration \
PGPASSFILE=/run/secrets/registry-relay-migration.pgpass \
pg_restore --exit-on-error --single-transaction --no-owner --no-acl \
  --role=relay_state_owner \
  --dbname=registry_relay \
  "$backup_dir/relay-consultation-state.dump"
```

The checksum command prints `OK`. The restore command must finish without an error before
bootstrap attestation starts.

After the restore, restore the matching config, artifact closure, PostgreSQL trust root, audit
HMAC material, and retained audit-pseudonym key material. Rerun
`registry-relay consultation bootstrap-state` with exactly the same role connections, owner,
chain-key epoch, lock keys, active key, deadline, and retention inputs recorded with the backup.
The command must report `installed_or_attested` and `identical`. A drift response means the
recovery set does not match; do not drop the Relay schemas or change bootstrap inputs to make it
pass.

For audited SnapshotExact, startup reconciles the database-authoritative active generation and
digest with the restored cache before opening a newer source generation. A missing, mismatched, or
unreadable retained cache generation keeps the dependent SnapshotExact publication unavailable.
Do not replace the database pointer, edit the cache, or open live source fallback to clear the
failure. Restore the matching coordinated set or republish a reviewed source input as a new
generation.

Relay startup and `/ready` attest the restored catalog, role grants, keyring, audit, quota,
materialization, and current serving fence. They do not prove that the recovery point includes
every write Relay previously acknowledged. If the backup might be stale, keep it offline and use
write-ahead-log replay or point-in-time recovery until the recovery point covers the last
acknowledged consultation and external audit watermark. If that cannot be proved, do not resume
the same consultation workload from the restore. Waiting for the 15-minute batch-child replay
window does not restore missing audit, quota, materialization, or keyring history.

## Restore Notary PostgreSQL data

Restore the dump into an empty, isolated database with no Notary traffic path. Provision the
restricted migration login, `NOLOGIN` owner role, and runtime role before the restore:

```sh
sha256sum -c "$backup_dir/notary-state.dump.sha256"
PGSERVICEFILE=/run/registry-notary/pg_service.conf \
PGSERVICE=registry_notary_migrator \
PGPASSFILE=/run/secrets/registry-notary-migrator.pgpass \
pg_restore --exit-on-error --single-transaction --no-owner --no-acl \
  --role=registry_notary_owner \
  --dbname=registry_notary \
  "$backup_dir/notary-state.dump"
registry-notary --config /etc/registry-notary/notary.yaml state install \
  --migration-url-env REGISTRY_NOTARY_POSTGRES_MIGRATOR_URL \
  --owner-role registry_notary_owner \
  --runtime-role registry_notary_runtime
registry-notary --config /etc/registry-notary/notary.yaml state doctor
```

The checksum command prints `OK`. The restore omits source-cluster ACLs and creates objects as the
new `NOLOGIN` owner. The installer safely rebinds role identities and reapplies the complete
compiled ACL baseline when the released schema still attests. It does not change ownership or
repair drift. The installer and doctor report schema version and supported PostgreSQL major
without printing database credentials. Restore the matching sensitive-state key before running
doctor. When preauthorization is enabled, doctor proves that the key is present and well formed,
but only a complete preauthorization canary proves that it decrypts state from this recovery point.

Keep Notary offline when the recovery point might predate an acknowledged write. Wait for every
possibly missing replay, nonce, evaluation, idempotency, preauthorization, and quota decision to
expire, and reconcile credential status before admitting traffic. A lost suspension or revocation
does not become safe through waiting. The
[Registry Notary PostgreSQL operations guide](https://docs.registrystack.org/products/registry-notary/postgresql-state-operations/)
defines the full stale-restore quarantine.

## Verify after restore

Start the deployment and run product checks:

```sh
registryctl start
registryctl doctor --profile local
```

The human-readable `doctor` report must contain no `startup_fail` findings. Add `--format json`
when another program needs the versioned report.
Run `registryctl smoke` when the generated deployment includes Relay. For a Notary deployment,
run `registry-notary doctor --config <your-notary-config.yaml> --format json`, then exercise an
authenticated claim or credential journey appropriate to that deployment. `registryctl` does not
provide a product-scoped Notary smoke subcommand. When OID4VCI preauthorization is enabled,
complete an offer-to-credential journey before admitting traffic.
For a consultation-enabled Relay, require `/ready` to return `200`, then run one authenticated
consultation canary while downstream Notary traffic remains blocked. Global readiness can remain
`200` while an individual SnapshotExact execution rejects a stale snapshot, so the canary must
exercise every recovered materialization relied on by the deployment. Admit Notary traffic only
after both product recovery points and the canaries have been reconciled.

For a deployment that uses `config_trust`, verify that the restored anti-rollback state still
matches the configured trust anchor and signed bundle before the service is allowed to serve.
Run the command on the host or inside a container where the configured paths are available:

```sh
registry-relay config verify-bundle \
  --bundle-dir /etc/registry-relay/config/bundle \
  --anchor-path /etc/registry-relay/config/trust-anchor.json \
  --state-path /var/lib/registry-relay/config-state/antirollback.json
```

Use the equivalent `registry-notary config verify-bundle` command and Notary paths when
verifying a Notary bundle.
A rollback rejection means the restored anti-rollback file has seen a newer bundle sequence
than the bundle you are trying to boot.

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| Generated API keys no longer work after restore | `secrets/local.env` was regenerated or copied from the wrong project | Restore the original `secrets/local.env`, then restart the containers |
| Audit records cannot be correlated across the restore | The audit HMAC secret changed | Restore the prior `secrets/local.env`; if the old file is gone, document the break as a key-rotation event |
| Relay consultation bootstrap reports capability or keyring drift | The restore has different role object identifiers, schema ownership, bootstrap identity, or keyring lifecycle state | Keep traffic blocked and restore the matching complete recovery set; do not drop schemas or reinitialize the keyring |
| Relay is ready but the recovery point might be stale | Readiness attests the restored state plane but cannot infer missing acknowledged writes | Keep Relay and Notary traffic blocked and recover forward to the last acknowledged write and audit watermark |
| Relay is ready but one SnapshotExact canary is unavailable | The execution-time freshness bound failed, or the restored source, cache generation, content digest, and database publication pointer do not agree | Keep dependent traffic blocked and restore the coordinated recovery set or publish a reviewed new generation |
| Notary state doctor rejects the restore | The schema, runtime role, server major, write authority, durability settings, or configured sensitive-state key is invalid | Keep traffic blocked, restore the matching recovery set, and rerun `state install` and `state doctor` |
| Config bundle startup fails with a rollback code | The anti-rollback state records a newer accepted sequence | Boot a signed bundle with a higher sequence, or follow the break-glass process documented for config trust |

## Next

- [Run single-node Compose behind your reverse proxy](../single-node-compose-behind-proxy/)
- [Retention and persistent state](../retention-and-persistent-state/)
- [Upgrade and roll back a deployment](../upgrade-and-rollback/)
- [Harden a production deployment](../../security/hardening-checklist/)