Skip to content
Registry StackDocsLatest

Back up and restore a deployment

View as Markdown

Use this procedure before you recreate containers, move a host, or upgrade a single-node Registry Relay or Registry Notary deployment. Start with secrets/local.env because losing it rotates generated API credentials, audit HMAC secrets, and the local Notary issuer identity; then preserve product state directories, Redis data, config-trust files, and source inputs.

Use this for registryctl generated projects and for hand-written single-node Docker Compose deployments that keep state on the host running the containers. Use your platform’s snapshot, secret-manager, and database backup process for hosted, multi-node, or Kubernetes deployments.

This procedure preserves durable state. It does not preserve process-local state such as OpenID Connect (OIDC) JSON Web Key Set (JWKS) caches, source-adapter token caches, or in-flight OpenID for Verifiable Credential Issuance pre-authorized flows. Those flows are interrupted by restart and must be started again by the holder or upstream journey.

  • Know the project directory that contains compose.yaml, registryctl.yaml, secrets/, and state/.
  • Know whether the deployment includes Registry Notary and the registry-notary-redis service.
  • Stop write traffic or use a filesystem snapshot if you need an exact point-in-time copy of audit files.
  • Keep backups encrypted and access-controlled. secrets/local.env, audit files, config-trust state, and Redis replay data are security-sensitive.

Create a backup directory and copy secrets/local.env before copying other state:

Terminal window
backup_dir="backup/registry-stack-$(date +%Y%m%d)"
install -d -m 700 "$backup_dir"
cp -p secrets/local.env "$backup_dir/local.env"
sha256sum "$backup_dir/local.env" > "$backup_dir/local.env.sha256"
chmod 600 "$backup_dir/local.env" "$backup_dir/local.env.sha256"

Expected output is no output. The checksum file lets you confirm that a later restore put back the exact same secret file.

Back up config, source files, and product state

Section titled “Back up config, source files, and product state”

Back up the project manifest when present, Compose file, product config directories, source data, and host-mounted state:

Terminal window
backup_items=""
for item in registryctl.yaml compose.yaml relay notary state data; do
[ -e "$item" ] && backup_items="$backup_items $item"
done
[ -f .env ] && backup_items="$backup_items .env"
tar -cf "$backup_dir/config-source-state.tar" $backup_items
sha256sum "$backup_dir/config-source-state.tar" > "$backup_dir/config-source-state.tar.sha256"

Expected output is one checksum line in config-source-state.tar.sha256. The archive includes state/relay/ and state/notary/ when those directories exist. Those paths are where generated Compose projects mount Relay cache, optional file-audit directories, and config-trust anti-rollback state if you enable it with antirollback_state_path.

If your hand-written config uses different paths, include them in the archive:

  • Relay or Notary file-audit directories, such as /var/log/registry-relay/ or /var/log/registry-notary/.
  • config_trust.trust_anchor_path, config_trust.bundle_path, and config_trust.antirollback_state_path.
  • Source files, metadata manifests, signed bundles, and trust anchors that are mounted into the containers.

Generated Notary projects use Redis for replay and nonce state. The generated Redis service writes append-only files to the registry-notary-redis-data volume. Capture that volume, asking Redis to flush a snapshot first when the service is running:

Terminal window
redis_container="$(docker compose ps --all -q registry-notary-redis)"
if [ -n "$redis_container" ]; then
if [ "$(docker inspect "$redis_container" --format '{{.State.Running}}')" = "true" ]; then
docker compose exec -T registry-notary-redis redis-cli SAVE
fi
redis_volume="$(docker inspect "$redis_container" --format '{{range .Mounts}}{{if eq .Destination "/data"}}{{.Name}}{{end}}{{end}}')"
docker run --rm \
-v "$redis_volume:/redis:ro" \
-v "$PWD/$backup_dir:/backup" \
busybox tar -cf /backup/registry-notary-redis-data.tar -C /redis .
sha256sum "$backup_dir/registry-notary-redis-data.tar" > "$backup_dir/registry-notary-redis-data.tar.sha256"
fi

redis-cli SAVE prints OK when Redis wrote the snapshot. If the project has no Notary Redis service, the script block produces no backup archive.

On a new host, install Docker Compose and registryctl, create the project directory, and copy the backup directory into it. On the original host, stop the generated services before restoring files:

Terminal window
registryctl stop

Expected output comes from Docker Compose and shows the containers stopping or already stopped. Then restore secrets and host-mounted state:

Terminal window
sha256sum -c "$backup_dir/local.env.sha256"
install -d -m 700 secrets
cp -p "$backup_dir/local.env" secrets/local.env
sha256sum -c "$backup_dir/config-source-state.tar.sha256"
tar -xf "$backup_dir/config-source-state.tar"

Both checksum commands print OK for the named file. Do not generate a new secrets/local.env to replace a lost one unless you intend to rotate the deployment’s API credentials, audit HMAC secret, and local Notary issuer key. On a new host, compare .env with id -u and id -g. If REGISTRY_STACK_RUNTIME_UID or REGISTRY_STACK_RUNTIME_GID names the old host owner, edit those two values before starting containers so Relay and Notary can write the restored private state directories. If the archive was extracted by root or with numeric owners preserved, align restored state ownership and private permissions with the runtime identity:

Terminal window
runtime_uid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_UID" { print $2 }' .env)"
runtime_gid="$(awk -F= '$1 == "REGISTRY_STACK_RUNTIME_GID" { print $2 }' .env)"
if [ -n "$runtime_uid" ] && [ -n "$runtime_gid" ] && [ -d state ]; then
sudo chown -R "$runtime_uid:$runtime_gid" state
find state -type d -exec chmod 700 {} +
find state -type f -exec chmod 600 {} +
fi

Restore Redis before starting Notary if the backup includes registry-notary-redis-data.tar:

Terminal window
if [ -f "$backup_dir/registry-notary-redis-data.tar" ]; then
sha256sum -c "$backup_dir/registry-notary-redis-data.tar.sha256"
docker compose up -d registry-notary-redis
redis_container="$(docker compose ps -q registry-notary-redis)"
redis_volume="$(docker inspect "$redis_container" --format '{{range .Mounts}}{{if eq .Destination "/data"}}{{.Name}}{{end}}{{end}}')"
docker compose stop registry-notary-redis
docker run --rm \
-v "$redis_volume:/redis" \
-v "$PWD/$backup_dir:/backup:ro" \
busybox sh -c 'cd /redis && rm -rf ./* && tar -xf /backup/registry-notary-redis-data.tar'
fi

The checksum command prints OK. The temporary docker run command is quiet when the archive extracts successfully.

Start the deployment and run product checks:

Terminal window
registryctl start
registryctl doctor --profile local --format json
if [ -f registryctl.yaml ] && grep -q '^relay:' registryctl.yaml; then
registryctl smoke
fi
if [ -f registryctl.yaml ] && grep -q '^notary:' registryctl.yaml; then
registryctl notary smoke
fi

Expected output from doctor is a JSON report with no startup_fail findings. Run the Relay and Notary smoke checks only when registryctl.yaml declares the matching product section.

For a deployment that uses config_trust, verify that the restored anti-rollback state still matches the configured trust anchor and signed bundle before the service is allowed to serve. Run the command on the host or inside a container where the configured paths are available:

Terminal window
registry-relay config verify-bundle \
--bundle-dir /etc/registry-relay/config/bundle \
--anchor-path /etc/registry-relay/config/trust-anchor.json \
--state-path /var/lib/registry-relay/config-state/antirollback.json

Use the equivalent registry-notary config verify-bundle command and Notary paths when verifying a Notary bundle. A rollback rejection means the restored anti-rollback file has seen a newer bundle sequence than the bundle you are trying to boot.

SymptomCauseFix
Generated API keys no longer work after restoresecrets/local.env was regenerated or copied from the wrong projectRestore the original secrets/local.env, then restart the containers
Audit records cannot be correlated across the restoreThe audit HMAC secret changedRestore the prior secrets/local.env; if the old file is gone, document the break as a key-rotation event
Notary starts but replay checks behave like a fresh deploymentThe Redis volume was removed or restored emptyRestore registry-notary-redis-data.tar, or accept that in-flight replay and nonce state was lost
Config bundle startup fails with a rollback codeThe anti-rollback state records a newer accepted sequenceBoot a signed bundle with a higher sequence, or follow the break-glass process documented for config trust