Registry stack documentation: machine-readable Markdown.
Index of all pages: https://docs.registrystack.org/dev/llms.txt
Full corpus: https://docs.registrystack.org/dev/llms-full.txt

# Deploy Registry Coordinator

> Create the keys, database, and runtime file for Registry Coordinator, apply and serve its workflow packages, watch its metrics, and recover, erase, rotate, and restore what it holds.

An author has handed you Registry Coordinator packages and their digests. Serve those workflows
behind your TLS edge, then inspect, recover, erase, rotate, and restore them. The full rules are in
the [deployment runbook](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md) in the repository.

Authors package workflows in [Author a Coordinator workflow](../../configure/coordinator/).

## What you provide

- **PostgreSQL 17 or newer**, with one database and two login roles: a migration role that owns the
  schema and a runtime role that serves.
- **Your OpenID Connect issuer**, whose RFC 9068 access tokens (`typ: at+jwt`) callers present; each
  product connection is a client there.
- **A TLS terminator** in front of a service listener bound to a private address.
- **Secret files** for three keys, the two database URLs, and each connection credential, owned by
  the service user.
- **Durable storage** for the audit file.
- **The package directory and digest of each workflow**, up to sixteen in one deployment, with the
  author's runtime template.
- **The downstream policy the workflows need**, such as Base Registry Engine (BReg) access profiles, the Casework trusted
  producer, and Messaging templates, granting each connection identity only what it uses.

Install `coordinator` and `coordinatorctl` from one source version, at absolute paths, as
[Install and package](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#install-and-package) describes.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Deployment checklist. */}

## Create the keys, secrets, and roles

Coordinator uses three keys of 32 random bytes: the **payload state key** encrypts run state and
can gain versions; the **admission key** derives start identities and command keys and never
changes; the **audit key** derives the audit journal's keyed references. The runtime file names
each secret as `secret:file/<name>`, an owner-only file directly under the provider's `root`.

```sh
install -d -m 0700 /run/registry-coordinator/secrets
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/state-key-1 32)
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/admission-key 32)
(umask 077 && openssl rand -out /run/registry-coordinator/secrets/audit-key 32)
```

:::danger[A database without its keys cannot be read]
A restored database needs every state key version its rows used and the deployment's admission
key. Back up each key file under custody separate from the database.
:::

Write the runtime and migration database URLs, with `sslmode=require`, into owner-only files.
Coordinator verifies the database certificate against the system roots, or against
`database.trustedRootCertificateRef` for a private root. A
connection's `signingKeyRef` names a private JWK whose public half you register at the issuer, one
client and key per downstream authority. The runbook gives the commands and the SQL for the roles:
[Create the keys and secrets](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#create-the-keys-and-secrets) and
[Create the database and roles](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#create-the-database-and-roles).

`apply` and startup refuse a runtime role that is a superuser, bypasses row-level security, belongs
to the migration role, or can create objects in the schema. Allow 32 connections per process.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Create the database and roles. */}

## Write the runtime file

Start from the author's `runtime.yaml` and complete it. Keep the file outside the packages; it
holds secret references, never values.

| Key | What it sets |
| --- | --- |
| `namespace` | The deployment's SQL namespace, `coordinator_` followed by lowercase letters, digits, or `_`. |
| `secretProviders` | Where references resolve: `secret:file/<name>` under `file.root`, and `secret:env/<NAME>` only when `environment: {}` is present. |
| `eventSources` | Optional trusted producers whose signed events start runs, each with its `source` URI and an HMAC `signingKeyRef`. |
| `deployment.databaseId` | The deployment's durable identity; a mismatch refuses startup and apply. |
| `deployment.runtimeRole` | The runtime role, matching the role in `database.runtimeUrlRef`. |
| `deployment.packages` | One `root` and `expectedDigest` per workflow, up to sixteen. |
| `deployment.listener` | The service socket `bind`, with `tlsTermination: operator-controlled-upstream` behind your TLS terminator. |
| `deployment.metricsListener` | Optional private address that serves `/metrics`. |
| `deployment.authentication` | The issuer, audience, signing-key source, and caller policies. |
| `deployment.auditFile`, `auditKeyRef`, `stateKeys`, `activeStateKey`, `admissionKeyRef` | The audit file and the three keys. |

Key `authentication.policies` by verified client identity. A policy's `workflows` scopes its run
actions: `start`, `status`, `inspect`, `retry-same`, `cancel`, and `reconcile`. The
deployment-wide actions `doctor`, `restore-hold`, `release-restore-hold`, `release-admission-hold`,
`complete-execution-recovery`, and `retain` need `operator: true`. They act on every workflow at
once, so a policy that lists only deployment-wide actions omits `workflows`, and the configuration
check refuses one that sets it. Keep producers and operators in separate policies. A run belongs
to its starting token's `iss` and `sub`.

The [configuration reference](../../reference/coordinator-configuration/) lists every key and
bound, and [Configure the deployment](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#configure-the-deployment) covers each connection type.

### Let Casework wake waiting runs

A run waiting on a Casework review reads the result every `resultCheckSeconds`, from 60 to 86,400
and 900 by default. Each read is one request to Casework per waiting run. A `completion` block on
the Casework connection lets Casework wake the run sooner:

```yaml
connections:
  review:
    type: casework-review-producer
    producer: coordinator
    # baseUrl, profile, authorization as for any Casework connection
    completion:
      recipientBinding: coordinator-v1
      bearerTokenRef: secret:file/review-completion-token
    resultCheckSeconds: 900
```

- **`producer`** is the producer's key under the Casework project's `trustedProducers`. That
  entry's `completion.destinationId` names a destination, and its `completion.recipientBinding`
  repeats the binding below.
- **`recipientBinding`** is 1 to 256 visible ASCII characters, without spaces, and must differ
  between Casework connections; `check` refuses a duplicate.
- **`bearerTokenRef`** names a token you generate, of at most 4096 bytes:

  ```sh
  (umask 077 && openssl rand -hex 32 | tr -d '\n' > /run/registry-coordinator/secrets/review-completion-token)
  ```

  In the Casework runtime file, `reviewCompletionDestinations.<destinationId>` gives that
  destination its own reference to the same token as `bearerTokenRef`, not `auth` with a custom
  header, and `url: https://<coordinator-host>/v1/review-completions`;
  [Completion destinations](../casework-retention/#completion-destinations) describes those keys.
  Casework presents the token as a bearer token and the binding in a `registry-recipient-binding`
  header; Coordinator refuses a request that lacks either, or whose pair matches no connection,
  with 401 `authentication.refused`.

An event only brings a waiting run's next read forward; it never starts a run. Without a
`completion` block the route refuses every event, so lower `resultCheckSeconds` when events cannot
reach Coordinator.

A live run's binding includes `producer` but neither `completion` nor `resultCheckSeconds`. You
can change either, or rotate the token, while runs wait: write the new token on both sides and
restart each service. Changing `producer` is a binding change that startup refuses while runs use
it.

Check the file offline; it reads no credential and contacts no endpoint:

```sh
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml check
```

{/* Evidence: products/coordinator/DEPLOYMENT.md, Configure the deployment;
    products/coordinator/generated/runtime/runtime.schema.json, namespace, SecretProvidersConfig,
    EventSourceConfig, PrivateListenerConfig, TlsTermination, ReviewCompletionConfig,
    resultCheckSeconds;
    crates/registry-coordinator/src/deployment.rs, tls;
    crates/registry-coordinator/src/runtime.rs, valid_recipient_binding, binding_digest_names;
    crates/registry-coordinator/src/completion.rs, receive_review_completion,
    RECIPIENT_BINDING_HEADER. */}

## Apply and start

```sh
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml plan
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml apply
coordinatorctl --runtime-config /etc/registry-coordinator/runtime.yaml deployment-status
coordinator --runtime-config /etc/registry-coordinator/runtime.yaml
```

- `plan` reports, for each workflow, whether apply would `install`, `replace`, leave `unchanged`, or
  `remove` its package.
- `apply` is the only command that reads the migration URL. It verifies each pinned package,
  migrates the schema, grants the runtime role, and records each workflow's active package.
- `deployment-status` uses only the runtime credential and reports `status: checked` with each
  workflow's package digest and the `schemaVersion`.
- `coordinator` verifies the same things before it serves, and never migrates.

State schema version 4 is required, and an earlier layout is refused; settle that work with its
matching binary and start a fresh namespace and database identity.

Startup refuses when a runtime file change alters a live run's endpoint, identity, scopes, or task
authority. Restore the original binding, or investigate with `--recovery-only`, which admits and
dispatches nothing.

While the database is unreachable the service logs `store-unavailable`, answers `/ready` with 503,
and keeps running. SIGTERM stops it within 15 seconds.

With `deployment.listener.bind` at `127.0.0.1:8108`, probe `/ready`, then ask for the doctor
report:

```sh
curl -fsS http://127.0.0.1:8108/ready
coordinatorctl doctor --url https://coordinator.example.org --token-file /run/operator-token
```

`/ready` answers `{"ready":true}`. Pass a token by file path, never by value. In the doctor report,
`activationReady` and `state.auditReady` are `true`, the holds `false`, and `state.uncertain`
counts steps waiting for you.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Apply and start, Check health and run doctor;
    crates/registry-coordinator/src/store.rs, SCHEMA_VERSION;
    crates/registry-coordinator/src/service.rs, SHUTDOWN_GRACE. */}

### Codes you see

Each surface names a refusal in its own vocabulary. The service's standard error carries the raw
code: `store-unavailable` while the database is unreachable, `deployment.refused` or
`state-key-custody` when the service stops, and `live-binding-conflict` when startup finds a live
run's binding changed. `coordinatorctl` prefixes its codes with `coordinator.`, as in
`coordinator.deployment.refused`, `coordinator.deployment.workflow-unlisted`,
`coordinator.deployment.workflow-live-runs`, `coordinator.command.state-key-custody`, and
`coordinator.command.live-binding-conflict`. HTTP answers carry RFC 9457 problem documents; the
ones an operator acts on are:

| Code | Status | What to do |
| --- | --- | --- |
| `service.unavailable` | 503 | The database, audit stream, or token verifier is unavailable. Try again after it recovers. |
| `audit.response-unavailable` | 503 | The operation may have completed, but its audited result was not released. Inspect the run before repeating it. |
| `deployment.refused` | 409 | The activation, runtime file, or state key custody check failed. Run `deployment-status`. |
| `deployment.activation-changed` | 409 | An apply changed the active packages after the service started. Restart it with the runtime file apply used. |
| `binding.conflict`, `binding.live-conflict` | 409 | The runtime file changed connection settings a run uses. Restore them and restart, or investigate with `--recovery-only`. |
| `execution.held`, `admission.held` | 409 | The restore hold or the admission hold is in place. |
| `admission.recovery-only` | 409 | This instance runs with `--recovery-only` and admits no runs. Send starts to the ordinary service. |
| `restore.hold-required`, `restore.unresolved`, `restore.review-required`, `restore.lease-incompatible` | 409 | A restore step is out of order, or held work still needs review. |
| `reconcile.unsupported`, `reconcile.refused`, `reconcile.raced`, `retry.refused` | 409 | The held call cannot be reconciled or retried that way, or changed meanwhile. Inspect it again. |
| `receipt.expired` | 409 | The product no longer holds the original receipt. This is not evidence the command was never sent. |
| `run.payload-erased` | 409 | Retention erased the run's payloads; it can no longer be retried or reconciled. |

The [API reference](../../reference/apis/registry-coordinator/) lists every code.

{/* Evidence: crates/registry-coordinator/src/problem.rs, ProblemCode::http_status, detail;
    crates/registry-coordinator/src/authoring.rs, coordinator. code prefix;
    crates/registry-coordinator/tests/deployment_postgres.rs, coordinator.deployment.refused,
    coordinator.command.state-key-custody, coordinator.command.live-binding-conflict,
    coordinator.deployment.workflow-unlisted;
    products/coordinator/DEPLOYMENT.md, Add, upgrade, roll back and remove a workflow. */}

## Change the workflows

To add, upgrade, or roll back a workflow, change only its `packages` entry, then `plan`, `apply`,
and restart every service. Runs in flight keep the definition they started with. Dropping an entry
does not remove a workflow: `plan` and `apply` refuse with
`coordinator.deployment.workflow-unlisted` until you list it again or pass
`--remove-workflow <id>`, refused while the workflow has runs that have not ended. Workflows that
must not share connections belong in separate deployments. See [Add, upgrade, roll back and remove a workflow](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#add-upgrade-roll-back-and-remove-a-workflow).

Metrics are Prometheus text with no workflow, run, or principal label. Alert on `coordinator_dispatch_jobs` going absent as well as on its value; the
series list is in [Watch metrics](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#watch-metrics).

{/* Evidence: products/coordinator/DEPLOYMENT.md, Watch metrics. */}

## Inspect runs

Applications start runs through the API. Find the runs that need you and look at one:

```sh
coordinatorctl list --attention true \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl inspect --run <run-id> \
  --url https://coordinator.example.org --token-file /run/operator-token
```

`list` returns the 20 newest runs, or up to 100 with `--limit`, with a `nextCursor`. `inspect` shows
each step, every effect as `accepted` or `unknown`, and a wait's timing, never inputs, product
responses, or prepared commands.
[How Registry Coordinator works](../../explanation/how-coordinator-works/#how-do-you-inspect-and-recover-a-run)
explains what the states mean.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Admit and inspect work. */}

## Recover a run

Coordinator retries by itself only after a definite retryable answer; an uncertain answer waits
for you. Inspect, then choose one, with a `--reason` that
is an investigation reference holding no personal data:

- **`reconcile`** asks Messaging or Scheduling for its receipt of the original command. A BReg
  Action or a Casework request has no such receipt, so `reconcile` cannot settle it.
- **`retry-same`** resends the original command and key, when the product's recovery contract and
  the original deadline allow it.
- **`cancel`** takes the workflow's `onCancel` path where one exists, and otherwise stops future
  calls. It never undoes an effect a product accepted.

```sh
coordinatorctl reconcile --run <run-id> --reason incident-2026-10-12 \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl retry-same --run <run-id> --reason incident-2026-10-12 \
  --url https://coordinator.example.org --token-file /run/operator-token
```

:::danger[A fresh idempotency key is a fresh operation]
Starting the same work under a new key can repeat its effects. Settle an uncertain call with
`reconcile` or `retry-same`, which keep its key, and infer nothing from an ordinary record read.
:::

Some failures need a corrected workflow, not a retry: `mapping-invalid` (a call succeeded but the
mapping after it did not) and `invalid-command` (a computed input broke its operation's contract
before anything was sent; the status names its `failurePath`). See
[Recover safely](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#recover-safely).

{/* Evidence: products/coordinator/DEPLOYMENT.md, Recover safely. */}

## Erase ended payloads

```sh
coordinatorctl retain --before 2026-07-01T00:00:00Z --limit 100 \
  --url https://coordinator.example.org --token-file /run/operator-token
```

`retain` erases the payloads of runs that ended before the cutoff, at most 100 per call, keeping
tombstones of spent start and command identities, and never touches live or uncertain work. Take
the cutoff from your retention policy.

:::danger[Erasing cannot be undone]
An erased run can no longer be retried or reconciled. Removing a state key version before the data
written under it is erased makes those runs unreadable in the same way.
:::

{/* Evidence: crates/registry-coordinator/src/cli.rs, Retain. */}

## Rotate keys

- **Payload state key:** add a version under `deployment.stateKeys`, keep every earlier one, set
  `activeStateKey` to it, then `apply`, `deployment-status`, and restart every service.
- **Audit key:** point `deployment.auditKeyRef` at a new key, `check`, and restart. Keep the old key
  with the journal it wrote; references across the change do not join.
- **Admission key:** it cannot rotate. If exposed, stop admissions, let runs settle, and admit new
  work in a fresh deployment with a new `namespace`, `databaseId`, and key.

[Rotate and replace keys](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#rotate-and-replace-keys) gives each procedure in full.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Rotate and replace keys. */}

## Restore from a backup

A backup can predate an effect a product already accepted. Nothing holds a restored database by
itself: you place the hold, from an instance that cannot execute.

:::danger[Never start an ordinary worker on a restored database]
An ordinary `coordinator` resumes dispatch at once and resends prepared commands to products whose
state has moved on since the backup. First stop or fence the old service and every worker that can
reach its database or credentials. Start only `coordinator --recovery-only` until you release the
restore hold.
:::

1. Restore the database together with every state key version, the admission key, the audit key
   and stream, and the exact packages.
2. Start a maintenance instance with `coordinator --runtime-config <file> --recovery-only`. It
   serves authenticated recovery operations, admits no runs, takes no events, and never dispatches.
3. Place the hold with `coordinatorctl restore-hold`. It holds execution and, separately,
   admissions.
4. Inspect each held run and `reconcile` every command that may have been sent after the backup. A
   missing receipt does not prove the command was never received.
5. When retained evidence establishes the history, attest it with `complete-execution-recovery`.
6. Record the fencing and reconciliation evidence, then `release-restore-hold`. It is refused with
   `restore.unresolved` while held work is unresolved.
7. Stop the recovery-only instance, start the ordinary `coordinator`, and `retry-same` or `cancel`
   what remains.
8. Reopen admissions with `release-admission-hold` only once every spent key is established.

Each `--reason` and `--recovery-reference` is the investigation reference, holding no personal
data:

```sh
coordinatorctl restore-hold --reason <reference> \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl complete-execution-recovery --recovery-reference <reference> \
  --execution-history-complete --prior-deployment-fenced \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl release-restore-hold --reason <reference> \
  --url https://coordinator.example.org --token-file /run/operator-token
coordinatorctl release-admission-hold --recovery-reference <reference> \
  --admission-history-complete --prior-deployment-fenced \
  --url https://coordinator.example.org --token-file /run/operator-token
```

The attestation flags are audited operator statements, not proof. [Backup and restore](https://github.com/registrystack/registry-stack/blob/main/products/coordinator/DEPLOYMENT.md#backup-and-restore) gives the full rules and a rehearsal in
disposable databases.

{/* Evidence: products/coordinator/DEPLOYMENT.md, Backup and restore;
    crates/registry-coordinator/src/cli.rs, RestoreHold, CompleteExecutionRecovery,
    ReleaseRestoreHold, ReleaseAdmissionHold;
    crates/registry-coordinator/src/service.rs, recovery_only (worker poll skipped,
    check_event_ingress_keys); crates/registry-coordinator/src/http.rs, recovery_only
    (admissions refused); crates/registry-coordinator/src/store/recovery.rs, restore_hold and
    admissions_hold. */}

## Next

- [Registry Coordinator API](../../reference/apis/registry-coordinator/) lists every problem code.
- [Registry Coordinator configuration reference](../../reference/coordinator-configuration/) lists
  every runtime key and operation.