Deploy Registry Coordinator

An author has handed you Registry Coordinator packages and their digests. Serve those workflows behind your TLS edge, then inspect, recover, erase, rotate, and restore them. The full rules are in the deployment runbook in the repository.

Authors package workflows in Author a Coordinator workflow.

What you provide

• PostgreSQL 17 or newer, with one database and two login roles: a migration role that owns the schema and a runtime role that serves.
• Your OpenID Connect issuer, whose RFC 9068 access tokens (typ: at+jwt) callers present; each product connection is a client there.
• A TLS terminator in front of a service listener bound to a private address.
• Secret files for three keys, the two database URLs, and each connection credential, owned by the service user.
• Durable storage for the audit file.
• The package directory and digest of each workflow, up to sixteen in one deployment, with the author’s runtime template.
• The downstream policy the workflows need, such as Base Registry Engine (BReg) access profiles, the Casework trusted producer, and Messaging templates, granting each connection identity only what it uses.

Install coordinator and coordinatorctl from one source version, at absolute paths, as Install and package describes.

Create the keys, secrets, and roles

Coordinator uses three keys of 32 random bytes: the payload state key encrypts run state and can gain versions; the admission key derives start identities and command keys and never changes; the audit key derives the audit journal’s keyed references. The runtime file names each secret as secret:file/<name>, an owner-only file directly under the provider’s root.

install -d -m 0700 /run/registry-coordinator/secrets
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/state-key-1 32)
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/admission-key 32)
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/audit-key 32)

A database without its keys cannot be read

A restored database needs every state key version its rows used and the deployment’s admission key. Back up each key file under custody separate from the database.

Write the runtime and migration database URLs, with sslmode=require, into owner-only files. Coordinator verifies the database certificate against the system roots, or against database.trustedRootCertificateRef for a private root. A connection’s signingKeyRef names a private JWK whose public half you register at the issuer, one client and key per downstream authority. The runbook gives the commands and the SQL for the roles: Create the keys and secrets and Create the database and roles.

apply and startup refuse a runtime role that is a superuser, bypasses row-level security, belongs to the migration role, or can create objects in the schema. Allow 32 connections per process.

Write the runtime file

Start from the author’s runtime.yaml and complete it. Keep the file outside the packages; it holds secret references, never values.

Key	What it sets
namespace	The deployment’s SQL namespace, coordinator_ followed by lowercase letters, digits, or _.
secretProviders	Where references resolve: secret:file/<name> under file.root, and secret:env/<NAME> only when environment: {} is present.
eventSources	Optional trusted producers whose signed events start runs, each with its source URI and an HMAC signingKeyRef.
deployment.databaseId	The deployment’s durable identity; a mismatch refuses startup and apply.
deployment.runtimeRole	The runtime role, matching the role in database.runtimeUrlRef.
deployment.packages	One root and expectedDigest per workflow, up to sixteen.
deployment.listener	The service socket bind, with tlsTermination: operator-controlled-upstream behind your TLS terminator.
deployment.metricsListener	Optional private address that serves /metrics.
deployment.authentication	The issuer, audience, signing-key source, and caller policies.
deployment.auditFile, auditKeyRef, stateKeys, activeStateKey, admissionKeyRef	The audit file and the three keys.

Key authentication.policies by verified client identity. A policy’s workflows scopes its run actions: start, status, inspect, retry-same, cancel, and reconcile. The deployment-wide actions doctor, restore-hold, release-restore-hold, release-admission-hold, complete-execution-recovery, and retain need operator: true. They act on every workflow at once, so a policy that lists only deployment-wide actions omits workflows, and the configuration check refuses one that sets it. Keep producers and operators in separate policies. A run belongs to its starting token’s iss and sub.

The configuration reference lists every key and bound, and Configure the deployment covers each connection type.

Let Casework wake waiting runs

A run waiting on a Casework review reads the result every resultCheckSeconds, from 60 to 86,400 and 900 by default. Each read is one request to Casework per waiting run. A completion block on the Casework connection lets Casework wake the run sooner:

connections:
  review:
    type: casework-review-producer
    producer: coordinator
    # baseUrl, profile, authorization as for any Casework connection
    completion:
      recipientBinding: coordinator-v1
      bearerTokenRef: secret:file/review-completion-token
    resultCheckSeconds: 900

• producer is the producer’s key under the Casework project’s trustedProducers. That entry’s completion.destinationId names a destination, and its completion.recipientBinding repeats the binding below.

• recipientBinding is 1 to 256 visible ASCII characters, without spaces, and must differ between Casework connections; check refuses a duplicate.

• bearerTokenRef names a token you generate, of at most 4096 bytes:

  (umask 077 && openssl rand -hex 32 | tr -d '\n' > /run/registry-coordinator/secrets/review-completion-token)

  In the Casework runtime file, reviewCompletionDestinations.<destinationId> gives that destination its own reference to the same token as bearerTokenRef, not auth with a custom header, and url: https://<coordinator-host>/v1/review-completions; Completion destinations describes those keys. Casework presents the token as a bearer token and the binding in a registry-recipient-binding header; Coordinator refuses a request that lacks either, or whose pair matches no connection, with 401 authentication.refused.

An event only brings a waiting run’s next read forward; it never starts a run. Without a completion block the route refuses every event, so lower resultCheckSeconds when events cannot reach Coordinator.

A live run’s binding includes producer but neither completion nor resultCheckSeconds. You can change either, or rotate the token, while runs wait: write the new token on both sides and restart each service. Changing producer is a binding change that startup refuses while runs use it.

Check the file offline; it reads no credential and contacts no endpoint:

coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml check

Apply and start

coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml plan
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml apply
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml deployment-status
coordinator --runtime-config /etc/registry-coordinator/runtime.yaml

• plan reports, for each workflow, whether apply would install, replace, leave unchanged, or remove its package.
• apply is the only command that reads the migration URL. It verifies each pinned package, migrates the schema, grants the runtime role, and records each workflow’s active package.
• deployment-status uses only the runtime credential and reports status: checked with each workflow’s package digest and the schemaVersion.
• coordinator verifies the same things before it serves, and never migrates.

State schema version 4 is required, and an earlier layout is refused; settle that work with its matching binary and start a fresh namespace and database identity.

Startup refuses when a runtime file change alters a live run’s endpoint, identity, scopes, or task authority. Restore the original binding, or investigate with --recovery-only, which admits and dispatches nothing.

While the database is unreachable the service logs store-unavailable, answers /ready with 503, and keeps running. SIGTERM stops it within 15 seconds.

With deployment.listener.bind at 127.0.0.1:8108, probe /ready, then ask for the doctor report:

curl -fsS http://127.0.0.1:8108/ready
coordinatorctl doctor --url https://coordinator.example.org --token-file /run/operator-token

/ready answers {"ready":true}. Pass a token by file path, never by value. In the doctor report, activationReady and state.auditReady are true, the holds false, and state.uncertain counts steps waiting for you.

Codes you see

Each surface names a refusal in its own vocabulary. The service’s standard error carries the raw code: store-unavailable while the database is unreachable, deployment.refused or state-key-custody when the service stops, and live-binding-conflict when startup finds a live run’s binding changed. coordinatorctl prefixes its codes with coordinator., as in coordinator.deployment.refused, coordinator.deployment.workflow-unlisted, coordinator.deployment.workflow-live-runs, coordinator.command.state-key-custody, and coordinator.command.live-binding-conflict. HTTP answers carry RFC 9457 problem documents; the ones an operator acts on are:

Code	Status	What to do
service.unavailable	503	The database, audit stream, or token verifier is unavailable. Try again after it recovers.
audit.response-unavailable	503	The operation may have completed, but its audited result was not released. Inspect the run before repeating it.
deployment.refused	409	The activation, runtime file, or state key custody check failed. Run deployment-status.
deployment.activation-changed	409	An apply changed the active packages after the service started. Restart it with the runtime file apply used.
binding.conflict, binding.live-conflict	409	The runtime file changed connection settings a run uses. Restore them and restart, or investigate with --recovery-only.
execution.held, admission.held	409	The restore hold or the admission hold is in place.
admission.recovery-only	409	This instance runs with --recovery-only and admits no runs. Send starts to the ordinary service.
restore.hold-required, restore.unresolved, restore.review-required, restore.lease-incompatible	409	A restore step is out of order, or held work still needs review.
reconcile.unsupported, reconcile.refused, reconcile.raced, retry.refused	409	The held call cannot be reconciled or retried that way, or changed meanwhile. Inspect it again.
receipt.expired	409	The product no longer holds the original receipt. This is not evidence the command was never sent.
run.payload-erased	409	Retention erased the run’s payloads; it can no longer be retried or reconciled.

The API reference lists every code.

Change the workflows

To add, upgrade, or roll back a workflow, change only its packages entry, then plan, apply, and restart every service. Runs in flight keep the definition they started with. Dropping an entry does not remove a workflow: plan and apply refuse with coordinator.deployment.workflow-unlisted until you list it again or pass --remove-workflow <id>, refused while the workflow has runs that have not ended. Workflows that must not share connections belong in separate deployments. See Add, upgrade, roll back and remove a workflow.

Metrics are Prometheus text with no workflow, run, or principal label. Alert on coordinator_dispatch_jobs going absent as well as on its value; the series list is in Watch metrics.

Inspect runs

Applications start runs through the API. Find the runs that need you and look at one:

coordinatorctl list --attention true \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl inspect --run <run-id> \
  --url https://coordinator.example.org --token-file /run/operator-token

list returns the 20 newest runs, or up to 100 with --limit, with a nextCursor. inspect shows each step, every effect as accepted or unknown, and a wait’s timing, never inputs, product responses, or prepared commands. How Registry Coordinator works explains what the states mean.

Recover a run

Coordinator retries by itself only after a definite retryable answer; an uncertain answer waits for you. Inspect, then choose one, with a --reason that is an investigation reference holding no personal data:

• reconcile asks Messaging or Scheduling for its receipt of the original command. A BReg Action or a Casework request has no such receipt, so reconcile cannot settle it.
• retry-same resends the original command and key, when the product’s recovery contract and the original deadline allow it.
• cancel takes the workflow’s onCancel path where one exists, and otherwise stops future calls. It never undoes an effect a product accepted.

coordinatorctl reconcile --run <run-id> --reason incident-2026-10-12 \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl retry-same --run <run-id> --reason incident-2026-10-12 \
  --url https://coordinator.example.org --token-file /run/operator-token

A fresh idempotency key is a fresh operation

Starting the same work under a new key can repeat its effects. Settle an uncertain call with reconcile or retry-same, which keep its key, and infer nothing from an ordinary record read.

Some failures need a corrected workflow, not a retry: mapping-invalid (a call succeeded but the mapping after it did not) and invalid-command (a computed input broke its operation’s contract before anything was sent; the status names its failurePath). See Recover safely.

Erase ended payloads

coordinatorctl retain --before 2026-07-01T00:00:00Z --limit 100 \
  --url https://coordinator.example.org --token-file /run/operator-token

retain erases the payloads of runs that ended before the cutoff, at most 100 per call, keeping tombstones of spent start and command identities, and never touches live or uncertain work. Take the cutoff from your retention policy.

Erasing cannot be undone

An erased run can no longer be retried or reconciled. Removing a state key version before the data written under it is erased makes those runs unreadable in the same way.

Rotate keys

• Payload state key: add a version under deployment.stateKeys, keep every earlier one, set activeStateKey to it, then apply, deployment-status, and restart every service.
• Audit key: point deployment.auditKeyRef at a new key, check, and restart. Keep the old key with the journal it wrote; references across the change do not join.
• Admission key: it cannot rotate. If exposed, stop admissions, let runs settle, and admit new work in a fresh deployment with a new namespace, databaseId, and key.

Rotate and replace keys gives each procedure in full.

Restore from a backup

A backup can predate an effect a product already accepted. Nothing holds a restored database by itself: you place the hold, from an instance that cannot execute.

Never start an ordinary worker on a restored database

An ordinary coordinator resumes dispatch at once and resends prepared commands to products whose state has moved on since the backup. First stop or fence the old service and every worker that can reach its database or credentials. Start only coordinator --recovery-only until you release the restore hold.

1. Restore the database together with every state key version, the admission key, the audit key and stream, and the exact packages.
2. Start a maintenance instance with coordinator --runtime-config <file> --recovery-only. It serves authenticated recovery operations, admits no runs, takes no events, and never dispatches.
3. Place the hold with coordinatorctl restore-hold. It holds execution and, separately, admissions.
4. Inspect each held run and reconcile every command that may have been sent after the backup. A missing receipt does not prove the command was never received.
5. When retained evidence establishes the history, attest it with complete-execution-recovery.
6. Record the fencing and reconciliation evidence, then release-restore-hold. It is refused with restore.unresolved while held work is unresolved.
7. Stop the recovery-only instance, start the ordinary coordinator, and retry-same or cancel what remains.
8. Reopen admissions with release-admission-hold only once every spent key is established.

Each --reason and --recovery-reference is the investigation reference, holding no personal data:

coordinatorctl restore-hold --reason <reference> \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl complete-execution-recovery --recovery-reference <reference> \
  --execution-history-complete --prior-deployment-fenced \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl release-restore-hold --reason <reference> \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl release-admission-hold --recovery-reference <reference> \
  --admission-history-complete --prior-deployment-fenced \
  --url https://coordinator.example.org --token-file /run/operator-token

The attestation flags are audited operator statements, not proof. Backup and restore gives the full rules and a rehearsal in disposable databases.

Next

• Registry Coordinator API lists every problem code.
• Registry Coordinator configuration reference lists every runtime key and operation.