Deploy and operate the Document Store

You have a checked Document Store project and its package digest, from Check your first Document Store project. Serve it behind your TLS edge, change it, and recover what it holds. Build documents and documentsctl from one source checkout, as the overview explains.

What you provide

• PostgreSQL 17 or newer, reached over TLS, with one database and two login roles: a migration role that owns the schema, and a runtime role that serves. documentsctl apply uses the first and grants the second what serving needs. Both connections verify the server certificate and its host name, so a local database needs a certificate too; an operator build refuses database.testOnlyPlaintext.
• Your OpenID Connect issuer. Every route except GET /health and GET /ready takes a bearer access token it signed; the token’s client, scopes, and actor kind select one access profile in the project. Access tokens lists what each token must carry.
• A content verifier: an HTTP endpoint that receives each stored version’s bytes and answers approved or rejected. It speaks the same protocol as the Base Registry Engine (BReg) external attachment verification.
• A TLS-terminating proxy in front of the listener, which binds 127.0.0.1:8116 in the starter.
• Optionally, an S3-compatible bucket for the bytes, used by this deployment alone and without object versioning. Without one, the bytes are held in PostgreSQL.

Provision PostgreSQL

Create the roles and the database:

CREATE ROLE documents_migration LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
  NOINHERIT NOBYPASSRLS;
CREATE ROLE documents_runtime LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
  NOINHERIT NOBYPASSRLS;
CREATE DATABASE documents;

Set each role’s password with \password documents_migration and \password documents_runtime in psql, which prompts for it instead of taking it in a statement.

In the documents database, hand the schema to the migration role:

REVOKE ALL ON DATABASE documents FROM PUBLIC;
GRANT CONNECT ON DATABASE documents TO documents_migration, documents_runtime;
ALTER SCHEMA public OWNER TO documents_migration;
REVOKE ALL ON SCHEMA public FROM PUBLIC;

Grant the runtime role nothing more. apply grants it what serving needs, and serve refuses a runtime role that holds authority over the schema. If the runtime role later loses USAGE on the schema or read access to the activation ledger, serve refuses to start and the operator commands answer documentsctl.activation.ledger-unreadable at database.runtimeUrlRef. Run apply again with the migration credential to restore its grants.

Write the runtime file

Start from the runtime.example.yaml that documentsctl init wrote and copy it to /etc/registry-documents/runtime.yaml. Every path in it is absolute, the file may not be writable by its group or by others, and every credential is a secret: reference to a declared provider. The choices that matter most:

Key	Decide
identity.databaseId	A name for the one database this deployment owns. The service refuses a database recorded for another.
package.root and package.expectedDigest	The package directory, and the digest documentsctl package printed.
database.trustedRootCertificateRef	The PEM bundle of the CA that signs the database certificate, when a private CA does. It replaces the system roots.
authentication.oidc	The issuer, the audience your tokens carry, every client in allowedClients, and jwksSource when the runtime should not fetch the issuer’s keys itself. See Access tokens.
storage.type	database or s3. The first upload pins the choice: a later start under another backend or bucket is refused.
storage.maximumDocumentBytes	The largest version the deployment takes: 16 MiB by default on both backends. On s3 you may raise it up to 64 MiB.
contentVerification	type: http with the verifier’s url, authorizationRef, and policyId. type: disabled approves everything and is accepted only on a development loopback listener.
destinations.hooks	One entry per destination id the project’s hooks name: the receiver’s url and its hmacSha256KeyRef. See Events.
retention.receiptRetentionDays	How long a retry with the same Idempotency-Key gets the same answer. Seven days is a placeholder for your retention schedule to approve.
retention.hookPayloadRetentionDays	How long an event waits to be delivered, from 1 to 30 days and 7 by default.
audit.rotateBytes and audit.retentionDays	For a file journal: the size at which the active file is sealed, 100 MiB by default, and how many days a sealed file is kept, 90 by default. Ship the records you must keep off the host before retentionDays deletes them.

Uploads, downloads, and verification attempts each hold a whole version in memory, and share one transfer budget: the larger of 256 MiB and storage.maximumDocumentBytes, which is 256 MiB at every ceiling the runtime accepts. Copies made while hashing, storing, and sending come on top of it, so give the host headroom beyond the budget and the process’s own needs. An upload or download that finds no room is refused at once with service.unavailable and Retry-After, and never waits. A verification attempt waits for room within its attempt timeout, then is retried later.

The configuration reference lists every key.

Access tokens

Issue each caller an RFC 9068 access token the runtime can verify and resolve to one access profile. Every token must have:

Part	Requirement
Header	typ: at+jwt (or application/at+jwt), alg RS256 or ES256, and a kid naming a key in the issuer’s key set
iss	Exactly authentication.oidc.issuer
aud	authentication.oidc.audience
exp	A time in the future; nbf, when present, is checked too
azp, or client_id when there is no azp	A client in allowedClients and in the requesterClients of the profile the caller should resolve to
scope (or the claim scopeClaim names)	Every scope in that profile’s requiredScopes, as a space-separated string or an array
registry_actor_kind	The profile’s actorKind: service or human
The profile’s principalClaim, usually sub	The caller’s identity, a non-empty string

A token that fails verification, lacks its principal, or carries a registry_actor_kind that is not a string answers 401 authentication.refused. A verified token that resolves to no profile, because of its client, a missing scope, or its actor kind, answers 403 profile.not-authorized. A token issued through a token exchange is refused unless authentication.oidc.assertionIssuers names the authority it came from.

By default the runtime fetches the issuer’s keys through its discovery document and follows its key rotation. When the runtime cannot reach the issuer, or a local test issuer publishes no discovery document, pin the keys instead:

authentication:
  oidc:
    jwksSource:
      type: static
      documentRef: secret:file/jwks.json

The secret holds a JSON Web Key Set of RSA or EC public keys, and every key needs its own non-empty kid: startup refuses a key without one, a repeated kid, and a symmetric key. The set is read once at startup and never rotates, so to change keys, replace the secret and restart serve.

Create the secret files

Create each file under the secretProviders.file.root directory as the user that runs documents, with no trailing newline. read -rs takes a value without echoing it or keeping it in shell history; paste postgres://documents_runtime:<password>@<database-host>:5432/documents at its prompt:

install -d -m 0700 /run/secrets/registry-documents
(umask 077; openssl rand -hex 32 | tr -d '\n' \
  > /run/secrets/registry-documents/documents-audit-key)
read -rs value
(umask 077; printf '%s' "$value" > /run/secrets/registry-documents/runtime-database-url)
unset value

Write the other files the runtime file names the same way: migration-database-url for documents_migration, content-verifier-authorization, each key the S3 storage names, and the signing key of each hook destination. The verifier file holds the token alone; the runtime sends it as Bearer <token>. Generate a hook key like the audit key, at least 32 bytes, and give the receiver the same bytes through its own secret store. Each file must be a regular file owned by the runtime user, mode 0400 or 0600, with one link.

serve never reads the migration credential; only apply connects with it. Write migration-database-url only on the host that runs apply.

Check, apply, and start

Check the file and the package before you deploy them:

documentsctl check --project "<project-directory>" --runtime-config /etc/registry-documents/runtime.yaml

checked: 0 errors, 0 warnings in 2 files

Add --environment to check ${VAR} values against the current environment. Then package the project, review what activation would change, and record the package:

documentsctl package "<project-directory>" --output /srv/registry-documents/package
documentsctl plan --runtime-config /etc/registry-documents/runtime.yaml

package digest: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c
change: activate

documentsctl apply --runtime-config /etc/registry-documents/runtime.yaml \
  --operator-reference "<change-reference>" --backup "<snapshot-id>"

package digest: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c
change: activate
restart documents serve to load the activated package

plan writes nothing. apply applies the schema, records the package digest as active, and grants the runtime role; applying the active package again records nothing. status shows the result:

documentsctl status --runtime-config /etc/registry-documents/runtime.yaml

active: sha256:ec47e52f55cd2071a8ba59ac0796816356caa521b6823e2fd54a1bc8a4e7971c
schema version: 100
runtime role mode: split

Start the service:

documents --runtime-config /etc/registry-documents/runtime.yaml serve

It logs JSON lines on standard error, starting with the Document Store is serving and the address. DOCUMENTS_LOG sets the level to error, warn, or info, the default; any other value stops documents serve with exit status 2. The log has no line per request: a request is logged only when something fails behind it, such as the database or the token verifier, or when its token lacks a scope its profile requires. A refused request is explained by its problem response, and every audited operation, such as an upload, a download, or a grant, is in the audit journal.

serve never applies the schema. Before it connects to the database, it refuses a runtime file that documentsctl check --project refuses against the package’s project, such as a hook destination destinations.hooks does not bind, with the same diagnostic on the same line. It refuses to start on a database with no active package, an active package other than the configured one, a schema from another release, or a runtime role that has lost its grants, and names every finding before it binds the socket. If its verification, event delivery, or erasure worker, or its cleanup pass, ends while it serves, it logs the error and exits with status 1, so run it under a supervisor that restarts it.

An upload wakes the verification worker of the same serve process as soon as the upload completes, so its version goes to the verifier at once. The worker also looks for due jobs every five seconds while idle, which picks up a job another instance or a documentsctl command enqueued, and one retried after a backoff.

Check that it answers:

curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8116/health
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8116/ready

200
200

GET /health is liveness: it answers 200 while the process serves. GET /ready is readiness: it answers 200 while the database carries this release’s schema, its activation ledger names the served package active under the current runtime role, and the audit journal accepts writes, and 503 service.unavailable otherwise, with the cause in the operational log. Point a load balancer’s health check at /ready. Once documentsctl apply activates another package, a process still serving the previous one answers /ready with 503 until it restarts onto the new package.

Every other route answers 401 authentication.refused without a valid token. The operator commands write their own audit journal beside the service’s.

Choose who may grant

Each document policy decides who can issue and revoke grants on its documents. The custodian rule names who holds a document placed for a named person: represented-principal makes that person the custodian, and institution keeps custody with the reserving service. Only the custodian grants. representedPrincipalGrants (permitted or refused) says whether the named person may grant when that person is the custodian. grantingProfiles lists the custodian profiles that may grant on documents they hold in custody; it may be empty, and a reserving service that is not the custodian never grants, even if its profile is listed. The starter’s citizen-delivery policy leaves it empty, so the issuing institution sees a delivered document’s metadata and status but cannot grant on it, list its grants, or read its bytes.

Change the project

A grant keeps its expiry; who may hold it follows the active package, so a change to a policy’s recipients applies to grants already issued. To change a policy, a profile, or a hook, edit documents.yaml and check it. package writes only into a directory that does not exist yet, so package into a new directory, set package.root to it and package.expectedDigest to the digest it prints, run plan and apply, then restart serve.

Changing only the runtime file needs only a restart, with one exception: a hook destination still holding undelivered events cannot be changed in place. Move a hook gives the procedure.

Changing the verifier’s url or policyId changes its binding. At the next start, every version verified under the old binding is suspended and verified again under the new one, so plan the change for a quiet period.

Replay a stuck verification

A verifier that cannot answer is retried for about seven days, then the job is dead-lettered and its version stays pending. List the stuck jobs, oldest first, with their generation:

documentsctl verification list --runtime-config /etc/registry-documents/runtime.yaml

b5631893-0b76-4871-97e6-e0934e60b1ab  generation 1  attempt 231  dead-lettered 2026-10-22T00:00:59Z

With none, it prints no dead-lettered content verification jobs. When the verifier answers again, return one job to pending:

documentsctl verification replay --runtime-config /etc/registry-documents/runtime.yaml \
  --version-id "<version-id>" --expected-generation "<generation>"

replayed: b5631893-0b76-4871-97e6-e0934e60b1ab is pending at generation 2

A generation other than the current one is refused, so two operators cannot replay the same job twice; a second replay of the same job is refused with error[documents.verification.not-dead-lettered] and exits 1. A version whose removal was requested, or whose bytes are being or have been erased, is refused too. No HTTP route lists or replays a job.

Removal and erasure

The custodian removes a document with POST /v1/documents/{documentId}/erasure. In one transaction the Document Store stops release, revokes every active grant, and marks the version erasure-pending, due after the policy’s cleanupGraceHours. That grace period is at least one hour, so you can still restore from a backup after a mistaken request. A sweep every minute erases what is due; a rejected version’s bytes are erased the same way. The record of the document, its version, and its grants stays.

What “erased” reaches depends on the storage:

Storage	After erasure
PostgreSQL	The content row is deleted. Its bytes stay on disk until PostgreSQL overwrites the pages or VACUUM FULL rewrites the table.
S3	The object is deleted, and a read must then answer 404 before the version reads erased.

On a versioned bucket that check passes on the delete marker while the old versions keep the bytes. Use an unversioned bucket, or a lifecycle rule that expires noncurrent versions. Backups, archived WAL, replicas, and bucket replicas keep erased bytes for as long as you keep them. Set their retention to the longest period your retention schedule allows erased content to survive; nothing in the runtime file controls it.

erasure status shows whether erasure runs and what waits:

documentsctl erasure status --runtime-config /etc/registry-documents/runtime.yaml

erasure: active
awaiting grace: 1
due: 0
dead-lettered: 0

An erasure job retries for about seven days before it dead-letters, and its version stays erasure-pending; the sweep never retries it on its own. List the dead-lettered jobs:

documentsctl erasure list --runtime-config /etc/registry-documents/runtime.yaml

no dead-lettered erasure jobs

Each job it lists has the same line as a dead-lettered verification: version id, generation, attempt, and time. Once the storage answers again, return one to pending:

documentsctl erasure replay --runtime-config /etc/registry-documents/runtime.yaml \
  --version-id "<version-id>" --expected-generation "<generation>"

A replay prints replayed: <version-id> is pending at generation <generation>. A version with no dead-lettered erasure is refused with error[documents.erasure.not-found], and a stale generation is refused, as for a verification.

To halt erasure without stopping the service, set cleanup.erasure: paused and restart. The service still serves, still takes removal requests, and logs a warning at startup. Due versions wait until you set it back to active and restart, and a replayed job waits with them.

Events

A project’s hooks tell another system that something changed. Each hook is an after-commit observer: once a change commits, it sends one signed event to a URL. In documents.yaml a hook names its trigger, the policies whose documents it covers, the projection fields it sends, and a handler.destinationId. The runtime file binds that id under destinations.hooks to the receiver’s url and hmacSha256KeyRef, in the same URL hook destination block Registry Scheduling reads, so the project carries no endpoint and no secret. The starter sends all three triggers for its three policies to one destination, owning-system-events.

policies is required and lists at least one policy the project declares. A hook hears nothing about a document held under a policy it does not list, so when policies serve different owning systems, give each system its own hooks and destination. The list is the only scope a hook has: it is reviewed with the package, and a hook cannot be routed by custodian.

Trigger	Sent when	Fields a hook may project
document-version.ready	A verdict makes a version ready that was not, including a version approved again after a change of verifier binding suspended it	documentId, versionId, versionDigest, policyId, policyVersion
document-version.rejected	A verdict rejects a version	The same five
document-grant.changed	A grant is issued or revoked, including each grant a removal revokes	The same five, plus grantId, change (issued or revoked), actorKind, and expiresAt

The body of an event is a Version 1 hook envelope. This is the starter’s grant hook reporting an issued grant, as observed, with identifiers and times replaced:

{
  "causation": { "hop": 0, "root": "<event-id>" },
  "data": {
    "actorKind": "service",
    "change": "issued",
    "documentId": "<document-id>",
    "expiresAt": "<grant-expiry>",
    "grantId": "<grant-id>",
    "versionId": "<version-id>"
  },
  "dataschema": "https://id.registrystack.org/schemas/documents/events/document-grant-changed.v1alpha1.schema.json",
  "id": "<event-id>",
  "source": "urn:registrystack:documents:civil-registry-documents",
  "subject": { "recordReference": "/v1/document-grants/<grant-id>", "recordRevision": 1 },
  "time": "<capture-time>",
  "type": "document-grant-changed"
}

• type is the hook’s id and dataschema names its trigger’s data schema, so two hooks on one trigger send events with different types and the same schema. source ends with the project id.
• data holds exactly the fields the hook projects. No event carries content, a label, a media type, a name, a subject, or anything else about the person a document describes.
• subject.recordRevision is the version’s or grant’s own revision. Every recorded change of the record raises it by one, so events about one record arrive with increasing revisions, and a receiver can drop one older than what it holds.
• actorKind is service or human: the kind of caller that issued or revoked the grant, or that requested the removal which revoked it. A version event has none, because its verdict is the verifier’s.
• Every grant event uses the one trigger document-grant.changed, and change says what happened: issued or revoked. To act on revocations only, filter on change: revoked. A grant outcome recorded later, such as an expiry, arrives as another change value on the same trigger, so a receiver reads the grant’s current state through the API on every event and does not treat an unfamiliar change as an error.
• expiresAt is the expiry fixed at issue. Reaching it sends nothing; a receiver that tracks grants reads the lapse from the issued event.
• A removal sends one revoked event for each grant it revokes, and the erasure that follows sends nothing. A repeated verdict, an issue replayed under its Idempotency-Key, a repeated revocation, and a repeated removal send nothing.

An event is captured in the same database transaction as the change it reports. A change that rolls back reports nothing, and if the event cannot be written, the verdict or grant change fails with it rather than commit unreported. Delivery runs after commit and never delays or undoes the change. A failed attempt is retried with a backoff that starts at 30 seconds and doubles up to an hour, until the destination’s maximumAttempts (8 by default, at most 20) are spent; the event is then dead-lettered. Each attempt is recorded in the audit journal as documents.hook-delivery. No documentsctl command lists, replays, or discards a delivery.

A receiver verifies the signature with its copy of the key, then reads what it needs through the API under its own token. An event grants nothing: it carries identifiers, and receiving one gives the receiver no standing in the Document Store. An observer cannot answer with a change of its own; the Document Store refuses every proposal, records it as documents.hook.proposal-unsupported, and changes nothing.

Move a hook

Hooks are part of the package: adding, removing, or changing one is a project change, applied with a new package as Change the project describes.

Where a hook sends is the runtime file’s, but an event waiting to be delivered keeps the destination binding it was captured under: the id, URL, key reference, timeout, and attempt budget. While any such event remains, documents serve refuses to start on a runtime file that changed or removed its destination, and exits with status 1:

documents: a retained hook delivery no longer matches its destination binding in destinations.hooks, or the hook delivery tables are unavailable

To move a hook to a new URL or key:

1. Add a destination under a new id in destinations.hooks, and leave the old entry exactly as it is.
2. Point the hook’s handler.destinationId at the new id, package and apply the project, and restart. New events go to the new destination; events already captured keep the old one.
3. Remove the old entry once nothing waits on it: every event captured for it has been delivered, has been dead-lettered, or has outlived retention.hookPayloadRetentionDays.

No command abandons pending deliveries, so a receiver that will never answer again still holds the old entry until its events are dead-lettered or expire.

Restore from a backup

Caution

A restore brings back grants revoked and documents removed after the backup. They must be repeated before you serve traffic again, and only their original callers can repeat them.

1. Set cleanup.erasure: paused and stop traffic at your edge.

2. Restore the database, and the audit file from the same point if you have it.

3. Start the service. It erases nothing while paused. Events the backup still records as undelivered are sent again, so a receiver may get an event it already holds; it recognises the repeat by its Idempotency-Key.

4. Compare the restored state with the audit file to find every revocation and removal newer than the backup. An operator holds no custodian or granting token, and no documentsctl command revokes a grant or removes a document, so ask each original caller to repeat its revocations and removals over HTTP. A repeated removal gets a fresh due time, so its bytes stay for another full cleanupGraceHours after the repeat.

5. On S3, if you restored the database and not the bucket, delete the objects uploaded after the backup: no row names them, so no sweep reaches them. Each object key is documents/<scope>/<sha256>, where <scope> is the lowercase hex SHA-256 of <identity.databaseId>/<object-id>, with the object id in lowercase hyphenated form. Compute the scope of every object_id in documents_staged_objects, list the keys under documents/, and delete every key whose scope none of them produces.

   Danger

   Run this step only on a bucket the deployment uses alone. Another deployment’s keys under the same prefix are indistinguishable from orphans, and this step deletes them.

6. Set cleanup.erasure: active and restart. The next sweep handles everything that became due.

Rehearse this drill before you rely on it.

Next

• Document Store configuration reference lists every key in both files.
• Document Store API operations describes each route.