Registry stack documentation: machine-readable Markdown. Index of all pages: https://docs.registrystack.org/dev/llms.txt Full corpus: https://docs.registrystack.org/dev/llms-full.txt # Deploy and operate the Document Store > Provide PostgreSQL, an issuer, and a content verifier for the Registry Document Store, apply and serve its package, replay stuck verifications, and handle removal, erasure, events, and restores. You have a checked Document Store project and its package digest, from [Check your first Document Store project](../../tutorials/first-documents/). Serve it behind your TLS edge, change it, and recover what it holds. Build `documents` and `documentsctl` from one source checkout, as the [overview](../../start/documents/#status) explains. ## What you provide - **PostgreSQL 17 or newer**, reached over TLS, with one database and two login roles: a migration role that owns the schema, and a runtime role that serves. `documentsctl apply` uses the first and grants the second what serving needs. Both connections verify the server certificate and its host name, so a local database needs a certificate too; an operator build refuses `database.testOnlyPlaintext`. - **Your OpenID Connect issuer.** Every route except `GET /health` and `GET /ready` takes a bearer access token it signed; the token's client, scopes, and actor kind select one access profile in the project. [Access tokens](#access-tokens) lists what each token must carry. - **A content verifier**: an HTTP endpoint that receives each stored version's bytes and answers approved or rejected. It speaks the same protocol as the Base Registry Engine (BReg) [external attachment verification](../../reference/breg-configuration/#external-attachment-verification). - **A TLS-terminating proxy** in front of the listener, which binds `127.0.0.1:8116` in the starter. - **Optionally, an S3-compatible bucket** for the bytes, used by this deployment alone and without object versioning. Without one, the bytes are held in PostgreSQL. {/* Evidence: crates/registry-documents/src/store.rs, PostgreSQL 17 refusal, connect_reference, and tls_connector; crates/registry-documents/src/config.rs, check_database; products/documents/RUNTIME-CONFIG.md, Keys and Startup; crates/registry-documents/src/verification.rs, activate_verifier; crates/registry-platform-objectstore/src/verifier.rs, HttpVerifier. */} ## Provision PostgreSQL Create the roles and the database: ```sql CREATE ROLE documents_migration LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOBYPASSRLS; CREATE ROLE documents_runtime LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOBYPASSRLS; CREATE DATABASE documents; ``` Set each role's password with `\password documents_migration` and `\password documents_runtime` in `psql`, which prompts for it instead of taking it in a statement. In the `documents` database, hand the schema to the migration role: ```sql REVOKE ALL ON DATABASE documents FROM PUBLIC; GRANT CONNECT ON DATABASE documents TO documents_migration, documents_runtime; ALTER SCHEMA public OWNER TO documents_migration; REVOKE ALL ON SCHEMA public FROM PUBLIC; ``` Grant the runtime role nothing more. `apply` grants it what serving needs, and `serve` refuses a runtime role that holds authority over the schema. If the runtime role later loses `USAGE` on the schema or read access to the activation ledger, `serve` refuses to start and the operator commands answer `documentsctl.activation.ledger-unreadable` at `database.runtimeUrlRef`. Run `apply` again with the migration credential to restore its grants. {/* Evidence: crates/registry-documents/src/activation.rs, apply (grant_runtime_role and stray_authority) and check_runtime; crates/registry-documents/src/activation.rs, LedgerUnreadable; crates/registry-documentsctl/src/operate.rs, failure; products/documents/RUNTIME-CONFIG.md, Startup; observed with documents 0.43.0-dev on 2026-10-22 against PostgreSQL 17. */} ## Write the runtime file Start from the `runtime.example.yaml` that `documentsctl init` wrote and copy it to `/etc/registry-documents/runtime.yaml`. Every path in it is absolute, the file may not be writable by its group or by others, and every credential is a `secret:` reference to a declared provider. The choices that matter most: | Key | Decide | | --- | ------ | | `identity.databaseId` | A name for the one database this deployment owns. The service refuses a database recorded for another. | | `package.root` and `package.expectedDigest` | The package directory, and the digest `documentsctl package` printed. | | `database.trustedRootCertificateRef` | The PEM bundle of the CA that signs the database certificate, when a private CA does. It replaces the system roots. | | `authentication.oidc` | The `issuer`, the `audience` your tokens carry, every client in `allowedClients`, and `jwksSource` when the runtime should not fetch the issuer's keys itself. See [Access tokens](#access-tokens). | | `storage.type` | `database` or `s3`. The first upload pins the choice: a later start under another backend or bucket is refused. | | `storage.maximumDocumentBytes` | The largest version the deployment takes: 16 MiB by default on both backends. On `s3` you may raise it up to 64 MiB. | | `contentVerification` | `type: http` with the verifier's `url`, `authorizationRef`, and `policyId`. `type: disabled` approves everything and is accepted only on a development loopback listener. | | `destinations.hooks` | One entry per destination id the project's hooks name: the receiver's `url` and its `hmacSha256KeyRef`. See [Events](#events). | | `retention.receiptRetentionDays` | How long a retry with the same `Idempotency-Key` gets the same answer. Seven days is a placeholder for your retention schedule to approve. | | `retention.hookPayloadRetentionDays` | How long an event waits to be delivered, from 1 to 30 days and 7 by default. | | `audit.rotateBytes` and `audit.retentionDays` | For a file journal: the size at which the active file is sealed, 100 MiB by default, and how many days a sealed file is kept, 90 by default. Ship the records you must keep off the host before `retentionDays` deletes them. | Uploads, downloads, and verification attempts each hold a whole version in memory, and share one transfer budget: the larger of 256 MiB and `storage.maximumDocumentBytes`, which is 256 MiB at every ceiling the runtime accepts. Copies made while hashing, storing, and sending come on top of it, so give the host headroom beyond the budget and the process's own needs. An upload or download that finds no room is refused at once with `service.unavailable` and `Retry-After`, and never waits. A verification attempt waits for room within its attempt timeout, then is retried later. The [configuration reference](../../reference/documents-configuration/) lists every key. {/* Evidence: products/documents/RUNTIME-CONFIG.md, Keys and Transfer memory; crates/registry-documents/src/config.rs, DATABASE_MAXIMUM_DOCUMENT_BYTES, S3_MAXIMUM_DOCUMENT_BYTES, MAXIMUM_HOOK_PAYLOAD_RETENTION_DAYS, AuditConfig, and ContentVerificationConfig; crates/registry-platform-audit/src/writer.rs, DEFAULT_AUDIT_ROTATE_BYTES and DEFAULT_AUDIT_RETAIN_DAYS; crates/registry-documents/src/transfer.rs, TRANSFER_BUFFER_BYTES and TransferBudget::new; crates/registry-documents/src/verification.rs, VerificationTransport. */} ## Access tokens Issue each caller an RFC 9068 access token the runtime can verify and resolve to one access profile. Every token must have: | Part | Requirement | | ---- | ----------- | | Header | `typ: at+jwt` (or `application/at+jwt`), `alg` `RS256` or `ES256`, and a `kid` naming a key in the issuer's key set | | `iss` | Exactly `authentication.oidc.issuer` | | `aud` | `authentication.oidc.audience` | | `exp` | A time in the future; `nbf`, when present, is checked too | | `azp`, or `client_id` when there is no `azp` | A client in `allowedClients` and in the `requesterClients` of the profile the caller should resolve to | | `scope` (or the claim `scopeClaim` names) | Every scope in that profile's `requiredScopes`, as a space-separated string or an array | | `registry_actor_kind` | The profile's `actorKind`: `service` or `human` | | The profile's `principalClaim`, usually `sub` | The caller's identity, a non-empty string | A token that fails verification, lacks its principal, or carries a `registry_actor_kind` that is not a string answers `401 authentication.refused`. A verified token that resolves to no profile, because of its client, a missing scope, or its actor kind, answers `403 profile.not-authorized`. A token issued through a token exchange is refused unless `authentication.oidc.assertionIssuers` names the authority it came from. By default the runtime fetches the issuer's keys through its discovery document and follows its key rotation. When the runtime cannot reach the issuer, or a local test issuer publishes no discovery document, pin the keys instead: ```yaml authentication: oidc: jwksSource: type: static documentRef: secret:file/jwks.json ``` The secret holds a JSON Web Key Set of RSA or EC public keys, and every key needs its own non-empty `kid`: startup refuses a key without one, a repeated `kid`, and a symmetric key. The set is read once at startup and never rotates, so to change keys, replace the secret and restart `serve`. {/* Evidence: products/documents/RUNTIME-CONFIG.md, Access tokens and Signing keys; crates/registry-documents/src/config.rs, verifier_profile and DOCUMENTS_ACCESS_TOKEN_TYPE; crates/registry-documents/src/auth.rs, authenticate; crates/registry-documents-core/src/access.rs, AccessProfiles::resolve; crates/registry-platform-oidc/src/lib.rs, verify_access_token, match_client, and parse_static_jwks; crates/registry-platform-oidc/src/authorization_claims.rs, ClaimNames (registry_actor_kind). */} ## Create the secret files Create each file under the `secretProviders.file.root` directory as the user that runs `documents`, with no trailing newline. `read -rs` takes a value without echoing it or keeping it in shell history; paste `postgres://documents_runtime:@:5432/documents` at its prompt: ```bash install -d -m 0700 /run/secrets/registry-documents (umask 077; openssl rand -hex 32 | tr -d '\n' \ > /run/secrets/registry-documents/documents-audit-key) read -rs value (umask 077; printf '%s' "$value" > /run/secrets/registry-documents/runtime-database-url) unset value ``` Write the other files the runtime file names the same way: `migration-database-url` for `documents_migration`, `content-verifier-authorization`, each key the S3 storage names, and the signing key of each hook destination. The verifier file holds the token alone; the runtime sends it as `Bearer `. Generate a hook key like the audit key, at least 32 bytes, and give the receiver the same bytes through its own secret store. Each file must be a regular file owned by the runtime user, mode `0400` or `0600`, with one link. `serve` never reads the migration credential; only `apply` connects with it. Write `migration-database-url` only on the host that runs `apply`. {/* Evidence: crates/registry-platform-config/src/secrets.rs, file provider checks; crates/registry-platform-objectstore/src/verifier.rs, HttpVerifier authorization; crates/registry-platform-crypto/src/delivery_signature.rs, MIN_HMAC_SHA256_KEY_BYTES; crates/registry-documents/src/runtime.rs, connect_migration only in apply. */} ## Check, apply, and start Check the file and the package before you deploy them: ```sh documentsctl check --project "" --runtime-config /etc/registry-documents/runtime.yaml ``` ```text checked: 0 errors, 0 warnings in 2 files ``` Add `--environment` to check `${VAR}` values against the current environment. Then package the project, review what activation would change, and record the package: ```sh documentsctl package "" --output /srv/registry-documents/package documentsctl plan --runtime-config /etc/registry-documents/runtime.yaml ``` ```text package digest: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c change: activate ``` ```sh documentsctl apply --runtime-config /etc/registry-documents/runtime.yaml \ --operator-reference "" --backup "" ``` ```text package digest: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c change: activate restart documents serve to load the activated package ``` `plan` writes nothing. `apply` applies the schema, records the package digest as active, and grants the runtime role; applying the active package again records nothing. `status` shows the result: ```sh documentsctl status --runtime-config /etc/registry-documents/runtime.yaml ``` ```text active: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c schema version: 100 runtime role mode: split ``` Start the service: ```sh documents --runtime-config /etc/registry-documents/runtime.yaml serve ``` It logs JSON lines on standard error, starting with `the Document Store is serving` and the address. `DOCUMENTS_LOG` sets the level to `error`, `warn`, or `info`, the default; any other value stops `documents serve` with exit status 2. The log has no line per request: a request is logged only when something fails behind it, such as the database or the token verifier, or when its token lacks a scope its profile requires. A refused request is explained by its problem response, and every audited operation, such as an upload, a download, or a grant, is in the audit journal. `serve` never applies the schema. Before it connects to the database, it refuses a runtime file that `documentsctl check --project` refuses against the package's project, such as a hook destination `destinations.hooks` does not bind, with the same diagnostic on the same line. It refuses to start on a database with no active package, an active package other than the configured one, a schema from another release, or a runtime role that has lost its grants, and names every finding before it binds the socket. If its verification, event delivery, or erasure worker, or its cleanup pass, ends while it serves, it logs the error and exits with status 1, so run it under a supervisor that restarts it. An upload wakes the verification worker of the same `serve` process as soon as the upload completes, so its version goes to the verifier at once. The worker also looks for due jobs every five seconds while idle, which picks up a job another instance or a `documentsctl` command enqueued, and one retried after a backoff. Check that it answers: ```sh curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8116/health curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8116/ready ``` ```text 200 200 ``` `GET /health` is liveness: it answers `200` while the process serves. `GET /ready` is readiness: it answers `200` while the database carries this release's schema, its activation ledger names the served package active under the current runtime role, and the audit journal accepts writes, and `503 service.unavailable` otherwise, with the cause in the operational log. Point a load balancer's health check at `/ready`. Once `documentsctl apply` activates another package, a process still serving the previous one answers `/ready` with `503` until it restarts onto the new package. Every other route answers `401 authentication.refused` without a valid token. The operator commands write their own audit journal beside the service's. {/* Evidence: products/documents/README.md, Operate; products/documents/RUNTIME-CONFIG.md, Startup; crates/registry-documentsctl/src/operate.rs; crates/registry-documents/src/main.rs, initialize_logging; crates/registry-documents/src/runtime.rs, operational_log_level; crates/registry-documents/src/audit.rs, module documentation; crates/registry-documents/src/runtime.rs, prepare_with, check_against_project, drain_until_stopped, and WORKER_IDLE_POLL; crates/registry-documents/src/service.rs, finalize and verification_wake; crates/registry-documents/src/http.rs, Readiness and ready, proven by the readiness tests in crates/registry-documents/tests/postgres_documents.rs; observed with documentsctl and documents 0.43.0-dev on 2026-10-22, paths and references replaced. */} ## Choose who may grant Each document policy decides who can issue and revoke grants on its documents. The `custodian` rule names who holds a document placed for a named person: `represented-principal` makes that person the custodian, and `institution` keeps custody with the reserving service. Only the custodian grants. `representedPrincipalGrants` (`permitted` or `refused`) says whether the named person may grant when that person is the custodian. `grantingProfiles` lists the custodian profiles that may grant on documents they hold in custody; it may be empty, and a reserving service that is not the custodian never grants, even if its profile is listed. The starter's `citizen-delivery` policy leaves it empty, so the issuing institution sees a delivered document's metadata and status but cannot grant on it, list its grants, or read its bytes. ## Change the project A grant keeps its expiry; who may hold it follows the active package, so a change to a policy's recipients applies to grants already issued. To change a policy, a profile, or a hook, edit `documents.yaml` and check it. `package` writes only into a directory that does not exist yet, so package into a new directory, set `package.root` to it and `package.expectedDigest` to the digest it prints, run `plan` and `apply`, then restart `serve`. Changing only the runtime file needs only a restart, with one exception: a hook destination still holding undelivered events cannot be changed in place. [Move a hook](#move-a-hook) gives the procedure. Changing the verifier's `url` or `policyId` changes its binding. At the next start, every version verified under the old binding is suspended and verified again under the new one, so plan the change for a quiet period. {/* Evidence: crates/registry-documents/src/verification.rs, module documentation and sweep_bindings; crates/registry-documents/src/service.rs, release by grant reads the active policy; crates/registry-documents/src/service/grants.rs, a grant stores only its expiry; crates/registry-platform-config/src/package.rs, an existing output directory is refused; observed with documentsctl 0.43.0-dev on 2026-10-22. */} ## Replay a stuck verification A verifier that cannot answer is retried for about seven days, then the job is dead-lettered and its version stays `pending`. List the stuck jobs, oldest first, with their generation: ```sh documentsctl verification list --runtime-config /etc/registry-documents/runtime.yaml ``` ```text b5631893-0b76-4871-97e6-e0934e60b1ab generation 1 attempt 231 dead-lettered 2026-10-22T00:00:59Z ``` With none, it prints `no dead-lettered content verification jobs`. When the verifier answers again, return one job to pending: ```sh documentsctl verification replay --runtime-config /etc/registry-documents/runtime.yaml \ --version-id "" --expected-generation "" ``` ```text replayed: b5631893-0b76-4871-97e6-e0934e60b1ab is pending at generation 2 ``` A generation other than the current one is refused, so two operators cannot replay the same job twice; a second replay of the same job is refused with `error[documents.verification.not-dead-lettered]` and exits 1. A version whose removal was requested, or whose bytes are being or have been erased, is refused too. No HTTP route lists or replays a job. {/* Evidence: crates/registry-documents/src/verification.rs, dead_letters, replay, and ReplayRefusal; crates/registry-documentsctl/src/lib.rs, View::DeadLetters and View::Replayed; observed with documentsctl 0.43.0-dev on 2026-10-22 against a job dead-lettered for the capture. */} ## Removal and erasure The custodian removes a document with `POST /v1/documents/{documentId}/erasure`. In one transaction the Document Store stops release, revokes every active grant, and marks the version `erasure-pending`, due after the policy's `cleanupGraceHours`. That grace period is at least one hour, so you can still restore from a backup after a mistaken request. A sweep every minute erases what is due; a rejected version's bytes are erased the same way. The record of the document, its version, and its grants stays. What "erased" reaches depends on the storage: | Storage | After erasure | | ------- | ------------- | | PostgreSQL | The content row is deleted. Its bytes stay on disk until PostgreSQL overwrites the pages or `VACUUM FULL` rewrites the table. | | S3 | The object is deleted, and a read must then answer 404 before the version reads `erased`. | On a versioned bucket that check passes on the delete marker while the old versions keep the bytes. Use an unversioned bucket, or a lifecycle rule that expires noncurrent versions. Backups, archived WAL, replicas, and bucket replicas keep erased bytes for as long as you keep them. Set their retention to the longest period your retention schedule allows erased content to survive; nothing in the runtime file controls it. `erasure status` shows whether erasure runs and what waits: ```sh documentsctl erasure status --runtime-config /etc/registry-documents/runtime.yaml ``` ```text erasure: active awaiting grace: 1 due: 0 dead-lettered: 0 ``` An erasure job retries for about seven days before it dead-letters, and its version stays `erasure-pending`; the sweep never retries it on its own. List the dead-lettered jobs: ```sh documentsctl erasure list --runtime-config /etc/registry-documents/runtime.yaml ``` ```text no dead-lettered erasure jobs ``` Each job it lists has the same line as a dead-lettered verification: version id, generation, attempt, and time. Once the storage answers again, return one to pending: ```sh documentsctl erasure replay --runtime-config /etc/registry-documents/runtime.yaml \ --version-id "" --expected-generation "" ``` A replay prints `replayed: is pending at generation `. A version with no dead-lettered erasure is refused with `error[documents.erasure.not-found]`, and a stale generation is refused, as for a verification. To halt erasure without stopping the service, set `cleanup.erasure: paused` and restart. The service still serves, still takes removal requests, and logs a warning at startup. Due versions wait until you set it back to `active` and restart, and a replayed job waits with them. {/* Evidence: products/documents/README.md, Retention and erasure; products/documents/RUNTIME-CONFIG.md, Keys; crates/registry-documents/src/erasure.rs, cleanup_once, dead_letters, report, and replay; observed with documentsctl 0.43.0-dev on 2026-10-22. */} ## Events A project's `hooks` tell another system that something changed. Each hook is an after-commit observer: once a change commits, it sends one signed event to a URL. In `documents.yaml` a hook names its `trigger`, the `policies` whose documents it covers, the `projection` fields it sends, and a `handler.destinationId`. The runtime file binds that id under `destinations.hooks` to the receiver's `url` and `hmacSha256KeyRef`, in the same URL hook destination block Registry Scheduling reads, so the project carries no endpoint and no secret. The starter sends all three triggers for its three policies to one destination, `owning-system-events`. `policies` is required and lists at least one policy the project declares. A hook hears nothing about a document held under a policy it does not list, so when policies serve different owning systems, give each system its own hooks and destination. The list is the only scope a hook has: it is reviewed with the package, and a hook cannot be routed by custodian. | Trigger | Sent when | Fields a hook may project | | ------- | --------- | ------------------------- | | `document-version.ready` | A verdict makes a version ready that was not, including a version approved again after a change of verifier binding suspended it | `documentId`, `versionId`, `versionDigest`, `policyId`, `policyVersion` | | `document-version.rejected` | A verdict rejects a version | The same five | | `document-grant.changed` | A grant is issued or revoked, including each grant a removal revokes | The same five, plus `grantId`, `change` (`issued` or `revoked`), `actorKind`, and `expiresAt` | The body of an event is a Version 1 hook envelope. This is the starter's grant hook reporting an issued grant, as observed, with identifiers and times replaced: ```json { "causation": { "hop": 0, "root": "" }, "data": { "actorKind": "service", "change": "issued", "documentId": "", "expiresAt": "", "grantId": "", "versionId": "" }, "dataschema": "https://id.registrystack.org/schemas/documents/events/document-grant-changed.v1alpha1.schema.json", "id": "", "source": "urn:registrystack:documents:civil-registry-documents", "subject": { "recordReference": "/v1/document-grants/", "recordRevision": 1 }, "time": "", "type": "document-grant-changed" } ``` - `type` is the hook's id and `dataschema` names its trigger's data schema, so two hooks on one trigger send events with different types and the same schema. `source` ends with the project id. - `data` holds exactly the fields the hook projects. No event carries content, a label, a media type, a name, a subject, or anything else about the person a document describes. - `subject.recordRevision` is the version's or grant's own revision. Every recorded change of the record raises it by one, so events about one record arrive with increasing revisions, and a receiver can drop one older than what it holds. - `actorKind` is `service` or `human`: the kind of caller that issued or revoked the grant, or that requested the removal which revoked it. A version event has none, because its verdict is the verifier's. - Every grant event uses the one trigger `document-grant.changed`, and `change` says what happened: `issued` or `revoked`. To act on revocations only, filter on `change: revoked`. A grant outcome recorded later, such as an expiry, arrives as another `change` value on the same trigger, so a receiver reads the grant's current state through the API on every event and does not treat an unfamiliar `change` as an error. - `expiresAt` is the expiry fixed at issue. Reaching it sends nothing; a receiver that tracks grants reads the lapse from the `issued` event. - A removal sends one `revoked` event for each grant it revokes, and the erasure that follows sends nothing. A repeated verdict, an issue replayed under its `Idempotency-Key`, a repeated revocation, and a repeated removal send nothing. An event is captured in the same database transaction as the change it reports. A change that rolls back reports nothing, and if the event cannot be written, the verdict or grant change fails with it rather than commit unreported. Delivery runs after commit and never delays or undoes the change. A failed attempt is retried with a backoff that starts at 30 seconds and doubles up to an hour, until the destination's `maximumAttempts` (8 by default, at most 20) are spent; the event is then dead-lettered. Each attempt is recorded in the audit journal as `documents.hook-delivery`. No `documentsctl` command lists, replays, or discards a delivery. A receiver verifies the signature with its copy of the key, then reads what it needs through the API under its own token. An event grants nothing: it carries identifiers, and receiving one gives the receiver no standing in the Document Store. An observer cannot answer with a change of its own; the Document Store refuses every proposal, records it as `documents.hook.proposal-unsupported`, and changes nothing. {/* Evidence: crates/registry-documents/src/hooks.rs, DocumentEvent::subject, projected, proposal_refusal, INITIAL_BACKOFF_MS, MAXIMUM_BACKOFF_MS, and record_audit; crates/registry-documents-core/src/project.rs, HookTrigger::fields; crates/registry-platform-config/src/blocks.rs, UrlHookDestinationConfig; products/documents/generated/events/; products/documents/contracts/recorded-decisions.yaml, DOCUMENTS-DEC-27 to DOCUMENTS-DEC-33; observed with documents 0.43.0-dev on 2026-10-22: a ready event, an issued grant event, and the revoked event of a removal. */} ### Move a hook Hooks are part of the package: adding, removing, or changing one is a project change, applied with a new package as [Change the project](#change-the-project) describes. Where a hook sends is the runtime file's, but an event waiting to be delivered keeps the destination binding it was captured under: the id, URL, key reference, timeout, and attempt budget. While any such event remains, `documents serve` refuses to start on a runtime file that changed or removed its destination, and exits with status 1: ```text documents: a retained hook delivery no longer matches its destination binding in destinations.hooks, or the hook delivery tables are unavailable ``` To move a hook to a new URL or key: 1. Add a destination under a new id in `destinations.hooks`, and leave the old entry exactly as it is. 2. Point the hook's `handler.destinationId` at the new id, package and apply the project, and restart. New events go to the new destination; events already captured keep the old one. 3. Remove the old entry once nothing waits on it: every event captured for it has been delivered, has been dead-lettered, or has outlived `retention.hookPayloadRetentionDays`. No command abandons pending deliveries, so a receiver that will never answer again still holds the old entry until its events are dead-lettered or expire. {/* Evidence: products/documents/contracts/recorded-decisions.yaml, DOCUMENTS-DEC-31; crates/registry-platform-hooks/src/delivery/service.rs, verify_retained_bindings; crates/registry-documents/tests/postgres_hooks.rs, a_destination_changed_under_a_pending_delivery_refuses_startup_until_it_drains; observed with documents 0.43.0-dev on 2026-10-22. */} ## Restore from a backup :::caution A restore brings back grants revoked and documents removed after the backup. They must be repeated before you serve traffic again, and only their original callers can repeat them. ::: 1. Set `cleanup.erasure: paused` and stop traffic at your edge. 2. Restore the database, and the audit file from the same point if you have it. 3. Start the service. It erases nothing while paused. Events the backup still records as undelivered are sent again, so a receiver may get an event it already holds; it recognises the repeat by its `Idempotency-Key`. 4. Compare the restored state with the audit file to find every revocation and removal newer than the backup. An operator holds no custodian or granting token, and no `documentsctl` command revokes a grant or removes a document, so ask each original caller to repeat its revocations and removals over HTTP. A repeated removal gets a fresh due time, so its bytes stay for another full `cleanupGraceHours` after the repeat. 5. On S3, if you restored the database and not the bucket, delete the objects uploaded after the backup: no row names them, so no sweep reaches them. Each object key is `documents//`, where `` is the lowercase hex SHA-256 of `/`, with the object id in lowercase hyphenated form. Compute the scope of every `object_id` in `documents_staged_objects`, list the keys under `documents/`, and delete every key whose scope none of them produces. :::danger Run this step only on a bucket the deployment uses alone. Another deployment's keys under the same prefix are indistinguishable from orphans, and this step deletes them. ::: 6. Set `cleanup.erasure: active` and restart. The next sweep handles everything that became due. Rehearse this drill before you rely on it. {/* Evidence: products/documents/README.md, Restore drill; crates/registry-documents/src/content.rs, S3_KEY_PREFIX and object scope; crates/registry-documentsctl/src/lib.rs, no revoke or removal command. */} ## Next - [Document Store configuration reference](../../reference/documents-configuration/) lists every key in both files. - [Document Store API operations](../../reference/apis/documents/) describes each route.