Back up and restore a deployment

View as Markdown

Use this procedure before you recreate containers, move a host, or upgrade a single-node Registry Relay or Registry Notary deployment. Start with secrets/local.env because losing it rotates generated API credentials, audit HMAC secrets, and the local Notary issuer identity; then preserve product state directories, every enabled product’s complete PostgreSQL database, config-trust files, and source inputs.

This page remains draft until complete backup and restore pass against the exact independently verified 1.0 candidate and its pinned standalone Solmara topology.

When to use this

Use this for registryctl generated projects and for hand-written single-node Docker Compose deployments that keep state on the host running the containers. Use your platform’s snapshot, secret-manager, and database backup process for hosted, multi-node, or Kubernetes deployments.

This procedure preserves durable state. It does not preserve process-local state such as OpenID Connect (OIDC) JSON Web Key Set (JWKS) caches or Relay source-integration token caches. A restart can still interrupt an in-flight OpenID for Verifiable Credential Issuance browser journey even when its bounded preauthorization state survives in PostgreSQL.

Before you start

Know the project directory that contains compose.yaml, registry-stack.yaml, secrets/, and state/.
Know whether Relay consultations or Registry Notary are enabled, which PostgreSQL database each product uses, and which role-provisioning record belongs to each database.
For a deployment with audited SnapshotExact materializations, remove dependent traffic, stop the Relay fence holder and every Notary caller, and prevent replacement processes from starting. Keep that topology quiesced while capturing source inputs, the Relay ingest cache, and the Relay consultation database at one coordinated recovery point.
For a deployment without audited SnapshotExact, stop write traffic or use a filesystem snapshot if you need an exact point-in-time copy of audit files.
Keep backups encrypted and access-controlled. secrets/local.env, audit files, config-trust state, PostgreSQL correctness state, and sensitive-state key versions are security-sensitive.

Back up generated secrets first

Create a backup directory and copy secrets/local.env before copying other state:

backup_dir="backup/registry-stack-$(date +%Y%m%d)"
install -d -m 700 "$backup_dir"
cp -p secrets/local.env "$backup_dir/local.env"
sha256sum "$backup_dir/local.env" > "$backup_dir/local.env.sha256"
chmod 600 "$backup_dir/local.env" "$backup_dir/local.env.sha256"

Expected output is no output. The checksum file lets you confirm that a later restore put back the exact same secret file.

Back up config, source files, and product state

Back up the project manifest when present, Compose file, product config directories, source data, and host-mounted state:

backup_items=""
for item in registry-stack.yaml compose.yaml relay notary state data; do
  [ -e "$item" ] && backup_items="$backup_items $item"
done
[ -f .env ] && backup_items="$backup_items .env"
tar -cf "$backup_dir/config-source-state.tar" $backup_items
sha256sum "$backup_dir/config-source-state.tar" > "$backup_dir/config-source-state.tar.sha256"

Expected output is one checksum line in config-source-state.tar.sha256. The archive includes state/relay/ and state/notary/ when those directories exist. Those paths are where generated Compose projects mount Relay cache, optional file-audit directories, and config-trust anti-rollback state if you enable it with antirollback_state_path.

When audited SnapshotExact is configured, run this archive command only after the topology is quiesced. Keep Relay and its Notary callers stopped until the matching Relay database backup finishes. A source archive, ingest-cache archive, and Relay database taken at independent live times do not form a recoverable materialization set.

If your hand-written config uses different paths, include them in the archive:

Relay or Notary file-audit directories, such as /var/log/registry-relay/ or /var/log/registry-notary/.
config_trust.trust_anchor_path, config_trust.bundle_path, and config_trust.antirollback_state_path.
Source files, metadata manifests, signed bundles, and trust anchors that are mounted into the containers.

Back up Relay consultation PostgreSQL state

Skip this section when Relay has no consultation configuration. The ordinary Relay ingest cache remains covered by the file archive.

The consultation state plane is a complete Relay-owned correctness boundary. Its relay_state_private and relay_state_api schemas contain durable consultation audit, completion and replay decisions, quotas, dispatch fencing, materialization publication state, and audit-pseudonym keyring metadata. They are separate from Notary’s registry_notary_private and registry_notary_api schemas. Never copy state tables between the products.

Remove traffic from Notary callers first, then stop the Relay fence holder and prevent another Relay process from starting. Keep the source inputs and Relay ingest cache unchanged while the database backup is captured. If Relay and Notary form one service, keep both products quiesced while their source, cache, and database backups are taken or use a platform snapshot that gives all stores one coordinated recovery point. A transactionally consistent dump of one live database does not make independently backed-up source, cache, or product databases share a recovery point.

For a database dedicated to Relay and an empty-database restore in the same PostgreSQL cluster, where the pre-created roles keep their object identifiers, back up the complete Relay database through the migration login:

PGSERVICEFILE=/run/registry-relay/pg_service.conf \
PGSERVICE=registry_relay_migration \
PGPASSFILE=/run/secrets/registry-relay-migration.pgpass \
pg_dump --format=custom --no-owner --no-acl \
  --role=relay_state_owner \
  --file="$backup_dir/relay-consultation-state.dump" \
  --dbname=registry_relay
sha256sum "$backup_dir/relay-consultation-state.dump" \
  > "$backup_dir/relay-consultation-state.dump.sha256"

If Relay shares a database with another owner, have the database administrator back up the complete database with an appropriately privileged backup identity. Do not narrow the backup to the two Relay schemas.

Expected output is one checksum line. Store the Relay release, config and signed artifact closure, PostgreSQL major, all five role names, the four bound role object identifiers, bootstrap inputs, keyring lifecycle settings, PostgreSQL trust root, and retained audit-pseudonym key material with the encrypted recovery record. Keep secret values outside the database dump.

For every audited SnapshotExact binding, the access-controlled recovery record must bind:

The source-input archive SHA-256 and the exact source revision.
The Relay ingest-cache archive SHA-256.
The complete Relay database backup SHA-256, including materialization active-pointer and history tables.
The exact active binding, publication sequence, generation, and restricted content digest.
The time or write-ahead-log position shared by the quiesced recovery set.

Hash and sign or independently attest that closed recovery record. Do not put restricted content digests, source paths, rows, credentials, or key material in general logs or a public release record. A list of unrelated artifact hashes is not evidence that the stores belong to one generation.

For recovery into another PostgreSQL cluster, use a physical backup or cluster-level managed snapshot that preserves the PostgreSQL role catalog and its object identifiers. Recreating the same role names does not preserve those identifiers, and Relay intentionally rejects the restored binding as capability drift. The Relay product documentation documents the schema and role boundary in full.

Back up PostgreSQL when Notary is present

Back up the complete Notary database at one consistent point. Do not back up individual correctness tables. Stop Notary writers for a quiesced backup, or use the database platform’s transactionally consistent snapshot workflow. Use a PostgreSQL service definition and password file so credentials do not appear in process arguments:

[registry_notary_migrator]
host=postgres.example.internal
port=5432
dbname=registry_notary
user=registry_notary_migrator
sslmode=verify-full
sslrootcert=/run/secrets/notary-postgres-ca.pem

The service name is operator-defined. The Registry Notary PostgreSQL operations guide provides the complete role-provisioning example and password-file requirements.

PGSERVICEFILE=/run/registry-notary/pg_service.conf \
PGSERVICE=registry_notary_migrator \
PGPASSFILE=/run/secrets/registry-notary-migrator.pgpass \
pg_dump --format=custom --no-owner --no-acl \
  --role=registry_notary_owner \
  --file="$backup_dir/notary-state.dump" \
  --dbname=registry_notary
sha256sum "$backup_dir/notary-state.dump" \
  > "$backup_dir/notary-state.dump.sha256"

Expected output is one checksum line. Preserve the application release, migration set, PostgreSQL major, reproducible owner and runtime role provisioning, recovery timestamp, and sensitive-state key version with the encrypted backup. Keep the key value in the secret manager, outside the database dump.

Restore files

On a new host, install Docker Compose and registryctl, create the project directory, and copy the backup directory into it. On the original host, stop the generated services before restoring files:

registryctl stop

Expected output comes from Docker Compose and shows the containers stopping or already stopped. Then restore secrets and host-mounted state:

sha256sum -c "$backup_dir/local.env.sha256"
install -d -m 700 secrets
cp -p "$backup_dir/local.env" secrets/local.env
sha256sum -c "$backup_dir/config-source-state.tar.sha256"
tar -xf "$backup_dir/config-source-state.tar"

Both checksum commands print OK for the named file. Do not generate a new secrets/local.env to replace a lost one unless you intend to rotate the deployment’s API credentials, audit HMAC secret, and local Notary issuer key. On a new host, compare .env with id -u and id -g. If REGISTRY_STACK_RUNTIME_UID or REGISTRY_STACK_RUNTIME_GID names the old host owner, edit those two values before starting containers so Relay and Notary can write the restored private state directories. If the archive was extracted by root or with numeric owners preserved, align restored state ownership and private permissions with the runtime identity:

runtime_uid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_UID" { print $2 }' .env)"
runtime_gid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_GID" { print $2 }' .env)"
if [ -n "$runtime_uid" ] && [ -n "$runtime_gid" ] && [ -d state ]; then
  sudo chown -R "$runtime_uid:$runtime_gid" state
  find state -type d -exec chmod 700 {} +
  find state -type f -exec chmod 600 {} +
fi

For audited SnapshotExact, do not start Relay after restoring the files. First verify that the source-input archive, ingest-cache archive, and Relay database dump match the same attested recovery record. The record’s active generation and restricted content digest must agree with the database active pointer and history and with the retained cache generation.

Restore Relay consultation PostgreSQL state

Skip this section when no Relay consultation dump or snapshot is present. Keep Relay and every Notary caller offline while restoring into an empty, isolated, writable PostgreSQL 16 through 18 primary.

For a same-cluster logical restore, retain the original owner, runtime, keyring-maintenance, and keyring-reader roles and their object identifiers. Restore the complete dump as the original owner. For a different cluster, restore the physical backup that includes the original role catalog. A logical restore into newly recreated role names is not a supported rebinding path.

sha256sum -c "$backup_dir/relay-consultation-state.dump.sha256"
PGSERVICEFILE=/run/registry-relay/pg_service.conf \
PGSERVICE=registry_relay_migration \
PGPASSFILE=/run/secrets/registry-relay-migration.pgpass \
pg_restore --exit-on-error --single-transaction --no-owner --no-acl \
  --role=relay_state_owner \
  --dbname=registry_relay \
  "$backup_dir/relay-consultation-state.dump"

The checksum command prints OK. The restore command must finish without an error before bootstrap attestation starts.

After the restore, restore the matching config, artifact closure, PostgreSQL trust root, audit HMAC material, and retained audit-pseudonym key material. Rerun registry-relay consultation bootstrap-state with exactly the same role connections, owner, chain-key epoch, lock keys, active key, deadline, and retention inputs recorded with the backup. The command must report installed_or_attested and identical. A drift response means the recovery set does not match; do not drop the Relay schemas or change bootstrap inputs to make it pass.

For audited SnapshotExact, startup reconciles the database-authoritative active generation and digest with the restored cache before opening a newer source generation. A missing, mismatched, or unreadable retained cache generation keeps the dependent SnapshotExact publication unavailable. Do not replace the database pointer, edit the cache, or open live source fallback to clear the failure. Restore the matching coordinated set or republish a reviewed source input as a new generation.

Relay startup and /ready attest the restored catalog, role grants, keyring, audit, quota, materialization, and current serving fence. They do not prove that the recovery point includes every write Relay previously acknowledged. If the backup might be stale, keep it offline and use write-ahead-log replay or point-in-time recovery until the recovery point covers the last acknowledged consultation and external audit watermark. If that cannot be proved, do not resume the same consultation workload from the restore. Waiting for the 15-minute batch-child replay window does not restore missing audit, quota, materialization, or keyring history.

Restore Notary PostgreSQL data

Restore the dump into an empty, isolated database with no Notary traffic path. Provision the restricted migration login, NOLOGIN owner role, and runtime role before the restore:

sha256sum -c "$backup_dir/notary-state.dump.sha256"
PGSERVICEFILE=/run/registry-notary/pg_service.conf \
PGSERVICE=registry_notary_migrator \
PGPASSFILE=/run/secrets/registry-notary-migrator.pgpass \
pg_restore --exit-on-error --single-transaction --no-owner --no-acl \
  --role=registry_notary_owner \
  --dbname=registry_notary \
  "$backup_dir/notary-state.dump"
registry-notary --config /etc/registry-notary/notary.yaml state install \
  --migration-url-env REGISTRY_NOTARY_POSTGRES_MIGRATOR_URL \
  --owner-role registry_notary_owner \
  --runtime-role registry_notary_runtime
registry-notary --config /etc/registry-notary/notary.yaml state doctor

The checksum command prints OK. The restore omits source-cluster ACLs and creates objects as the new NOLOGIN owner. The installer safely rebinds role identities and reapplies the complete compiled ACL baseline when the released schema still attests. It does not change ownership or repair drift. The installer and doctor report schema version and supported PostgreSQL major without printing database credentials. Restore the matching sensitive-state key before running doctor. When preauthorization is enabled, doctor proves that the key is present and well formed, but only a complete preauthorization canary proves that it decrypts state from this recovery point.

Keep Notary offline when the recovery point might predate an acknowledged write. Wait for every possibly missing replay, nonce, evaluation, idempotency, preauthorization, and quota decision to expire, and reconcile credential status before admitting traffic. A lost suspension or revocation does not become safe through waiting. The Registry Notary PostgreSQL operations guide defines the full stale-restore quarantine.

Verify after restore

Start the deployment and run product checks:

registryctl start
registryctl doctor --profile local

The human-readable doctor report must contain no startup_fail findings. Add --format json when another program needs the versioned report. Run registryctl smoke when the generated deployment includes Relay. For a Notary deployment, run registry-notary doctor --config <your-notary-config.yaml> --format json, then exercise an authenticated claim or credential journey appropriate to that deployment. registryctl does not provide a product-scoped Notary smoke subcommand. When OID4VCI preauthorization is enabled, complete an offer-to-credential journey before admitting traffic. For a consultation-enabled Relay, require /ready to return 200, then run one authenticated consultation canary while downstream Notary traffic remains blocked. Global readiness can remain 200 while an individual SnapshotExact execution rejects a stale snapshot, so the canary must exercise every recovered materialization relied on by the deployment. Admit Notary traffic only after both product recovery points and the canaries have been reconciled.

For a deployment that uses config_trust, verify that the restored anti-rollback state still matches the configured trust anchor and signed bundle before the service is allowed to serve. Run the command on the host or inside a container where the configured paths are available:

registry-relay config verify-bundle \
  --bundle-dir /etc/registry-relay/config/bundle \
  --anchor-path /etc/registry-relay/config/trust-anchor.json \
  --state-path /var/lib/registry-relay/config-state/antirollback.json

Use the equivalent registry-notary config verify-bundle command and Notary paths when verifying a Notary bundle. A rollback rejection means the restored anti-rollback file has seen a newer bundle sequence than the bundle you are trying to boot.

Troubleshooting

Symptom	Cause	Fix
Generated API keys no longer work after restore	`secrets/local.env` was regenerated or copied from the wrong project	Restore the original `secrets/local.env`, then restart the containers
Audit records cannot be correlated across the restore	The audit HMAC secret changed	Restore the prior `secrets/local.env`; if the old file is gone, document the break as a key-rotation event
Relay consultation bootstrap reports capability or keyring drift	The restore has different role object identifiers, schema ownership, bootstrap identity, or keyring lifecycle state	Keep traffic blocked and restore the matching complete recovery set; do not drop schemas or reinitialize the keyring
Relay is ready but the recovery point might be stale	Readiness attests the restored state plane but cannot infer missing acknowledged writes	Keep Relay and Notary traffic blocked and recover forward to the last acknowledged write and audit watermark
Relay is ready but one SnapshotExact canary is unavailable	The execution-time freshness bound failed, or the restored source, cache generation, content digest, and database publication pointer do not agree	Keep dependent traffic blocked and restore the coordinated recovery set or publish a reviewed new generation
Notary state doctor rejects the restore	The schema, runtime role, server major, write authority, durability settings, or configured sensitive-state key is invalid	Keep traffic blocked, restore the matching recovery set, and rerun `state install` and `state doctor`
Config bundle startup fails with a rollback code	The anti-rollback state records a newer accepted sequence	Boot a signed bundle with a higher sequence, or follow the break-glass process documented for config trust