Deploy Registry Messaging

An author has handed you a Registry Messaging package and its digest, and you want it serving. At the end of this page a messaging process serves that package against PostgreSQL behind your own TLS edge, sends through the providers you connected, and answers GET /ready, and you can inspect, retry, settle, and cancel its messages.

The package, its templates, and its access profiles stay with the author; Author a Messaging package covers that side. This page starts from the package directory.

What you provide

• One PostgreSQL 17 or newer database and two login credentials. The migration credential owns the schema, applies migrations, and records the active package. The runtime credential serves requests and runs the dispatch worker. The runtime requires TLS on both connections.
• One OpenID Connect issuer that issues RFC 9068 access tokens (typ: at+jwt) to the clients the package’s access profiles name, carrying the scopes those profiles require. Messaging verifies tokens and issues none.
• A TLS proxy or ingress in front of the listener. The process serves plain HTTP on a private or container-private address.
• A connection for each provider the package declares: an SMTP relay for email, and the base URL, credentials, and callback secret for each HTTP provider.
• Secrets the process can read, named by secret:file/<name> or secret:env/<NAME> references, never written into the runtime file.
• Durable storage for the audit journal, one directory owned by the runtime user with mode 0700.

Messaging needs no message broker and no cache: messages, the dispatch queue, attempts, and receipts all live in its own PostgreSQL database. It writes to no other product’s database.

Install or build the binaries

The first release eligible to publish Messaging is v0.38.0, after image-package onboarding is complete. For a published release that contains Messaging, choose <tag> from the latest release, download the messaging-<tag>-linux-amd64 and messagingctl-<tag>-linux-amd64 assets, and authenticate both through the shared SHA256SUMS procedure at https://github.com/registrystack/registry-stack/blob/<tag>/release/VERIFY.md. Install the exact pair from one tag:

tag="${TAG:?set TAG to a published tag that includes Registry Messaging}"
mkdir -p ~/.local/bin
for binary in messaging messagingctl; do
  install -m 0755 "${binary}-${tag}-linux-amd64" "$HOME/.local/bin/${binary}"
done
export PATH="$HOME/.local/bin:$PATH"
messaging --version
messagingctl --version

The install replaces either file already at that destination. Preserve the old pair first when you need a rollback path. Until a published release contains Messaging, or for another platform, build both binaries from a Registry Stack checkout:

cargo build --release --locked -p registry-messaging -p registry-messagingctl

On macOS, build through the runtime-library helper Author a Messaging package shows, so the shell can load the FIPS crypto library.

target/release/messaging is the runtime, with the serve subcommand. target/release/messagingctl is the operator tool; check, plan, apply, and status need no running runtime, and messages reads and changes messages through the database.

The release also publishes ghcr.io/registrystack/messaging:<tag>. The image carries /usr/local/bin/messaging and /usr/local/bin/messagingctl on Distroless cc-debian13 as the nonroot user. Its default command is messaging --runtime-config /etc/registry-messaging/runtime.yaml serve. It uses /var/lib/registry-messaging as its working directory and keeps the package, runtime configuration, and secrets outside the image. Mount those inputs read-only and provide writable persistent storage for the audit path. Authenticate the image digest through the release manifest and deploy that digest rather than the tag alone.

Provision PostgreSQL

Use PostgreSQL 17 or newer. Activation and startup refuse an older server. Create a migration role and a runtime role, and keep the migration credential off the serving host when no migration is in progress:

CREATE ROLE messaging_migration LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
  NOINHERIT NOBYPASSRLS PASSWORD '<migration-password>';
CREATE ROLE messaging_runtime LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
  NOINHERIT NOBYPASSRLS PASSWORD '<runtime-password>';
CREATE DATABASE messaging;

Messaging’s migrations create ordinary tables in the first schema of the connection’s search_path, public by default, and create no extension, schema, or role. As an administrator, in the messaging database:

REVOKE ALL ON DATABASE messaging FROM PUBLIC;
GRANT CONNECT ON DATABASE messaging TO messaging_migration, messaging_runtime;
ALTER SCHEMA public OWNER TO messaging_migration;
REVOKE ALL ON SCHEMA public FROM PUBLIC;
GRANT USAGE ON SCHEMA public TO messaging_runtime;
ALTER DEFAULT PRIVILEGES FOR ROLE messaging_migration IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO messaging_runtime;
ALTER DEFAULT PRIVILEGES FOR ROLE messaging_migration IN SCHEMA public
  GRANT USAGE, SELECT ON SEQUENCES TO messaging_runtime;

The default privileges cover every table messagingctl apply creates, because activation always runs as the migration role. When you reuse a database an earlier session already prepared, grant the same privileges on the tables and sequences that already exist.

Write the runtime file

The runtime file binds one package to one database, one issuer, one listener, and one connection per provider. Keep it outside the package, and put no credential in it:

apiVersion: registry.registrystack.org/messaging-runtime/v1alpha1
kind: MessagingRuntimeConfig
identity:
  databaseId: messaging-production
package:
  root: /srv/registry-messaging/package
  # The digest messagingctl check reported for the package you were handed.
  expectedDigest: sha256:beb567ec1a2813ac361cc1765151e398dbbb6e77bd886889769ed21c029f43f4
listener:
  bind: 10.42.0.7:8107
  tlsTermination: operator-controlled-upstream
  networkExposure: private-address
metricsListener:
  bind: 10.42.0.7:9107
secretProviders:
  file:
    root: /run/secrets/registry-messaging
database:
  runtimeUrlRef: secret:file/runtime-database-url
  migrationUrlRef: secret:file/migration-database-url
authentication:
  oidc:
    issuer: https://identity.example.test/realms/registry
    audience: urn:example:messaging
    allowedClients: [case-system, operations-console]
audit:
  path: /var/lib/registry-messaging/audit/audit.ndjson
  hashKeyRef: secret:file/messaging-audit-key
retention:
  payloadDays: 7
  recordDays: 90
  submissionReceiptDays: 7
providers:
  mail-relay:
    kind: smtp
    host: smtp.example.org
    tls: starttls
    authentication:
      usernameRef: secret:file/smtp-username
      passwordRef: secret:file/smtp-password
  sms-gateway:
    kind: http
    baseUrl: https://sms-gateway.example.org/v1/
    timeoutMilliseconds: 5000
    maximumResponseBytes: 65536
    concurrencyLimit: 4
    redirects: deny
    authentication:
      kind: static-authorization
      tokenRef: secret:file/sms-gateway-token
    callbackVerifier:
      kind: hmac-sha256-body
      header: x-signature
      encoding: hex
      secretRef: secret:file/sms-gateway-callback-key

Section	What it binds
identity	The stable operator-chosen id of the database this deployment owns
package	The absolute directory holding messaging.yaml, and optionally the one digest the runtime may load from it
listener	The public listener and the transport boundary it sits behind
metricsListener	The private socket that serves /metrics. Omit it to serve no metrics
secretProviders	The explicitly enabled file and environment secret providers
database	The runtime and migration connection URLs, and optionally trustedRootCertificateRef for a private CA
authentication.oidc	The issuer, audience, and admitted clients, and optionally scopeClaim, jwksSource, and assertionIssuers
audit	The per-process destination and the key for minimized references
retention	How long payloads, records, and submission receipts are kept
providers	One connection per provider the package declares, keyed by its id and tagged with the same kind
tlsTrustProfiles	Named PEM bundles an HTTP provider’s tlsTrustProfile selects

Every mapping is closed, so a misspelled key is refused with its path instead of falling back to a default. ${VAR}, ${VAR:-default}, and ${VAR:?message} substitute string values after YAML parsing. Keys and comments remain literal; substituted text cannot add keys or change structure. Expressions under Ref or Refs members and secret-provider declarations are refused. Diagnostics identify the field without revealing substituted values.

Listener and token rules

tlsTermination: operator-controlled-upstream is the production value: TLS terminates at your edge and the runtime stays plaintext behind it. development-loopback is direct plaintext and is refused on any non-loopback bind. networkExposure defaults to private-address, and container-private also permits an unspecified bind on a private container network. A public unicast address is refused under every combination. The metrics listener must be a concrete loopback or private address, and must not be the public listener’s socket.

allowedClients is never empty, and every client an access profile names must be listed. The scope claim is registry_scopes unless scopeClaim names another. Signing keys come from issuer discovery by default; declare jwksSource: {kind: static, documentRef: secret:file/<name>} when the runtime cannot reach the issuer’s discovery document. With discovery, serve fails at startup with the OIDC issuer could not be initialized when the issuer is unreachable.

Connect providers

A connection for a provider the package does not declare, or with another kind, is refused. A declared provider with no connection is not activated: startup logs a warning, and its messages fail with the attempt failure code provider-unconfigured without a send. Every credential, trust bundle, and callback secret is resolved once at startup, before either listener binds, and a provider that cannot be activated stops the runtime with an error naming the provider and the member, never a value.

An smtp connection takes host, tls (starttls on port 587 or implicit on port 465 by default), and optionally port, authentication, attemptTimeoutSeconds (1 to 60, default 30), trustedRootCertificateRef, and allowedPrivateCidrs. SMTP relays report no delivery, so an email message’s report is unavailable and submitted is the last status it reaches.

An http connection takes baseUrl, timeoutMilliseconds (at most 10000), maximumResponseBytes (at most 1 MiB), concurrencyLimit (at most the package’s), redirects: deny, and one authentication kind: none for a loopback http base URL only, basic, static-authorization, static-api-key, static-api-key-query, aws-sigv4, or oauth2-client-credentials. The runtime resolves the provider’s host and refuses a metadata, private, or other non-public address unless allowedPrivateCidrs names its network.

When the package declares receipts: callback for a provider, its connection needs a callbackVerifier, and give the provider this callback URL on your public edge:

kind	Keys	Callback URL
hmac-sha256-body	header, encoding (hex or base64), secretRef	/v1/provider-callbacks/<provider-id>
hmac-sha1-url-form	url, header, secretRef	/v1/provider-callbacks/<provider-id>
path-token	tokenRef	/v1/provider-callbacks/<provider-id>/<token>

For hmac-sha1-url-form, url is the external callback URL exactly as the provider was given it and signs, so a reverse proxy in front of the runtime must not change it.

Set retention

Key	Default	Bounds
payloadDays	7	1 to 30
recordDays	90	payloadDays to 3650
submissionReceiptDays	7	1 to recordDays

submissionReceiptDays is also the idempotency window: a repeated key whose stored receipt is older is refused with 410 idempotency.expired. A submission whose expiresAt falls after its payload period is refused. These values are defaults, not a jurisdictional recommendation, and the audit journal’s start record carries the values in force.

payloadDays and recordDays count from the moment a message reached a terminal state (delivered, failed, expired, or cancelled), not from acceptance, so a message waiting in a retry keeps its payload:

Period ends	What is erased
payloadDays after the terminal state	The rendered parts and the recipient contact. The message record stays
recordDays after the terminal state	The message record, with its attempts and receipts. The idempotency key stays spent, held only under the caller’s keyed pseudonym, and a repeat of it is refused with idempotency.expired
submissionReceiptDays after acceptance	The stored submission receipt. A repeat of its key is refused with idempotency.expired

A message that is queued, sending, or held in an unknown outcome is never erased, whatever its age. The runtime sweeps once at start and then hourly with the runtime credential. A sweep erases in batches of at most 1,000 of each kind, each batch its own transaction, and journals messaging.retention.erased with the counts for every batch that erased something. Erase expired data on demand runs the same sweep from the operator host.

Create the secret files

Create each file as the user that runs messaging, because the resolver refuses a file owned by anyone else:

install -d -m 0700 /run/secrets/registry-messaging
(umask 077; openssl rand -hex 32 | tr -d '\n' \
  > /run/secrets/registry-messaging/messaging-audit-key)
(umask 077; printf '%s' 'postgresql://messaging_runtime:<runtime-password>@db.example.org:5432/messaging' \
  > /run/secrets/registry-messaging/runtime-database-url)
(umask 077; printf '%s' 'postgresql://messaging_migration:<migration-password>@db.example.org:5432/messaging' \
  > /run/secrets/registry-messaging/migration-database-url)

Write the provider secrets the same way: here smtp-username, smtp-password, sms-gateway-token, and sms-gateway-callback-key. Each file must be a regular file owned by the effective user, with mode 0400 or 0600 and exactly one link. The bytes are used as written, so write them without a trailing newline. The audit key needs at least 32 bytes, which 64 hexadecimal characters satisfy. A reference that cannot resolve is refused by name, and the resolved bytes never appear in a message or a log.

Preserve the audit reference key

The key derives minimized caller and recipient references, including the caller pseudonym that scopes spent idempotency keys. Keep the same key across upgrades. Rotating it changes that identity and frees previously spent keys under the new pseudonym. The stream is not signed or hash-chained; ship records to append-only storage if you require external tamper evidence.

Check the runtime file

messagingctl check --runtime-config checks the runtime file and its package offline, before any database or issuer is reached:

messagingctl check --runtime-config /etc/registry-messaging/runtime.yaml

ok: /etc/registry-messaging/runtime.yaml
package digest: sha256:beb567ec1a2813ac361cc1765151e398dbbb6e77bd886889769ed21c029f43f4
listener: 127.0.0.1:8107
metrics listener: 127.0.0.1:9107
template: appointment-reminder 1 (email)
  sample en: renders
  sample fr: renders
template: appointment-reminder-sms 1 (sms)
  sample en: 1 segment(s), gsm7 85 units
access profile: case-notices (sender)
access profile: operations (operator)
retention: payload 7 days, record 90 days, submission receipt 7 days

That run used a loopback listener. When package.expectedDigest names another package, check, plan, apply, and serve all refuse before reaching a database:

error[config.refused] package.expectedDigest: the Messaging package at package.expectedDigest does not satisfy the shared package envelope: package.expectedDigest is sha256:0000000000000000000000000000000000000000000000000000000000000000 but the package at package.root is sha256:beb567ec1a2813ac361cc1765151e398dbbb6e77bd886889769ed21c029f43f4; deploy the pinned package or update package.expectedDigest
  next: Correct the member the path names, then rerun messagingctl check.

Plan, apply, inspect, and serve

Run the read-only plan from the operator host before the first deployment and before every upgrade:

messagingctl plan --runtime-config /etc/registry-messaging/runtime.yaml

plan connects with the runtime credential and writes nothing. It reports the configured package and active package digests, whether the database belongs to identity.databaseId, schema versions still to apply, and whether changes are pending. Review that report, then activate with the migration credential:

messagingctl apply --runtime-config /etc/registry-messaging/runtime.yaml

apply is the only command that changes the activation ledger. Under one PostgreSQL advisory lock, it applies pending schema versions, binds an unclaimed database to identity.databaseId, activates the verified package, grants the runtime role its required access, and records the activation as one transaction. A second apply waits for the lock instead of racing it. Applying an already active package with no pending schema versions and current runtime grants succeeds without a change: apply exits 0, reports change: "none" and applied: false, and records no activation. When the runtime role’s grants have gone stale, apply restores them and records a new activation.

Read the active state and activation history with the runtime credential:

messagingctl status --runtime-config /etc/registry-messaging/runtime.yaml

Then start the runtime with the runtime credential:

messaging --runtime-config /etc/registry-messaging/runtime.yaml serve

The runtime logs JSON lines on standard error, so a stdout audit destination carries audit entries alone. MESSAGING_LOG selects error, warn, or info, and any other value exits 2. Once both listeners are bound, it logs:

{"timestamp":"…","level":"INFO","fields":{"message":"serving Registry Messaging","listener":"127.0.0.1:8107","metrics_listener":true}}

GET /health answers 200 with an empty body while the process serves. GET /ready answers 200 while the database carries every expected migration and its package ledger names the package this process serves active and its audit writer is healthy, and 503 service.unavailable otherwise. Neither takes a token, and both carry Cache-Control: no-store.

On SIGTERM or SIGINT the runtime shuts down gracefully and exits 0: the worker claims no more messages and finishes the sends already in flight, a retention sweep under way finishes the batch it is erasing and starts no other, and both listeners stop accepting connections and finish the requests they hold. A message accepted but not yet claimed stays queued for the next start. Give the process time to finish a send before your service manager kills it, because a send cut short can leave its message unknown. A retention batch cut short rolls back, and the next start erases it and whatever the stopped sweep left. messagingctl dev does not drain this way: Ctrl-C stops its runtime at once.

Change the active package

serve refuses to start unless the package on disk has the digest the ledger names active, so an edited file never changes what a running deployment sends. To roll out a new package, replace the package directory, update package.expectedDigest if you set it, review messagingctl plan, run messagingctl apply, confirm the result with messagingctl status, and restart the runtime.

A recorded package stops the old one from restarting

Once apply records a new digest, every runtime that restarts on the old package directory refuses to start. Record a package only when every serving host has the new directory in place, and restart them together.

A restart against a package the ledger does not name fails with both digests:

messaging: the Messaging package on disk (sha256:41afc8ee…) is not the package the ledger names active (sha256:beb5…); record it with messagingctl apply, then restart

Each accepted message keeps the rendering and dispatch policy it had when it was accepted, so a package change affects only messages submitted after the restart.

A runtime still serving the old package answers GET /ready with 503 from the moment the new digest is recorded, and logs a warning naming the restart. A load balancer that checks /ready stops routing to it until it restarts onto the new package.

Operate messages

messagingctl messages reads messages through the runtime credential and never shows a recipient. list prints the newest first, and --status and --limit narrow it:

messagingctl messages list --runtime-config /etc/registry-messaging/runtime.yaml

<message-id> queued (dispatch queued) email transactional generation 1 attempt 1 accepted <timestamp> updated <timestamp>

show prints one message, its dispatch state and delivery report, and each attempt. Set MESSAGE_ID to an identifier from the list:

messagingctl messages show --runtime-config /etc/registry-messaging/runtime.yaml "$MESSAGE_ID"

message: <message-id>
status: queued (dispatch queued, report unavailable)
channel: email via transactional
to: email (redacted)
template: appointment-reminder 1
correlation id: case-4711
access profile: case-notices
generation: 1 attempt: 1
accepted: <timestamp>
expires: <timestamp>
updated: <timestamp>
attempt 1.1: transient started <timestamp> finished <timestamp>

This message’s first attempt could not reach its SMTP relay, so the attempt was transient and the message waits in queued for its next retry. The status the API reports, and what dispatch and report each mean, are in Registry Messaging API.

Retry, settle, and cancel

Each action decides on the message’s dispatch state, previews without --apply, and writes its request and outcome to its own audit stream once applied:

Command	Applies to dispatch state	Effect
messages retry	failed	Queues the message again under a new generation
messages settle --outcome sent	unknown	Marks the message submitted
messages settle --outcome not-sent	unknown	Queues the message again under a new generation
messages cancel	queued	Cancels the message

A submitted message the provider reported undelivered has the status failed but the dispatch state submitted, so it is not retried. An action on any other state is refused with exit 1:

error[message.not-eligible] MESSAGE_ID: message <message-id> has dispatch state queued; retry applies only to a message whose dispatch state is failed
  next: Run messagingctl messages show MESSAGE_ID and take the action its dispatch state allows.

Cancelling previews first:

messagingctl messages cancel --runtime-config /etc/registry-messaging/runtime.yaml "$MESSAGE_ID"

cancel <message-id>: queued -> cancelled
preview: run again with --apply to change the message

messagingctl messages cancel --runtime-config /etc/registry-messaging/runtime.yaml "$MESSAGE_ID" --apply

cancel <message-id>: queued -> cancelled
applied

A queued message waiting to retry after an attempt that may have reached the provider cannot be cancelled, because its first send may already have arrived. The preview and --apply both refuse it with exit 1 and message.dispatch-started, the code the HTTP API answers with.

A cancelled message is final: a caller cancelling it again over HTTP receives 409 message.terminal.

When an applied action does not finish, the error code says whether the message changed:

Code	Exit	Meaning
message.changed	1	The message changed between the preview and the write, and nothing was applied. Run messages show and decide again
audit.unavailable	3	The audit destination refused the request record, and nothing was changed
audit.unconfirmed	3	The action was applied, and its audit outcome record could not be written. Do not apply it again; restore the audit destination
message.outcome-unknown	3	The action may have been applied. Run messages show before acting again
database.unavailable	3	The database could not be reached or refused the write

Settle an unknown outcome

A message is unknown when a send may have reached the provider and nobody can tell whether it did: the connection broke after the request left, the attempt ran out of time, or the provider’s answer could not be read. Under the sender profile’s default onUncertain: hold, the worker stops there and sends nothing more until you settle it. Under onUncertain: retry, it sends again while the message has attempts and time left, then stops there, even when a later attempt failed or was refused, because a later failure does not show that the earlier send did not arrive.

Before settling, find out from the provider whether it took the message: its console or its logs, searched by the time of the attempt messages show lists. Settle sent when it did, and not-sent when it did not.

Settling not-sent can send the message twice

--outcome not-sent queues the message for another send. If the provider did take the first one, the recipient receives it twice. When you cannot establish the answer, weigh a duplicate against a missed message for this recipient before choosing.

Erase expired data on demand

messagingctl retention erase-expired runs the retention sweep from the operator host with the migration credential, counting every period back from --before. Without --apply it only reports what is due:

messagingctl retention erase-expired --runtime-config /etc/registry-messaging/runtime.yaml \
  --before 2026-09-25T00:00:00Z

before: 2026-09-25T00:00:00.000Z
payloads due: 0
records due: 0
submission receipts due: 0
run again with --apply to erase

Erasing cannot be undone

--apply deletes rendered parts, recipient contacts, and message records for good. Run the preview with the same --before first and read its counts.

messagingctl retention erase-expired --runtime-config /etc/registry-messaging/runtime.yaml \
  --before 2026-09-25T00:00:00Z --apply

before: 2026-09-25T00:00:00.000Z
payloads erased: 0
records erased: 0
submission receipts erased: 0

The command erases by the same rules as the runtime’s own sweep, in batches of at most 1,000 of each kind. Each batch is one transaction under one advisory lock, so the command and the runtime’s sweep never erase at once, and each batch is journaled as messaging.retention.erased with the operator tool as its actor. The first batch is journaled even when it erased nothing, so every applied run leaves a record. The counts printed are the totals across batches.

A run that fails partway keeps what it already erased: the batches committed before the failure stay erased and journaled, and the next run continues from there. When the failure leaves the last batch’s outcome unknown (retention.outcome-unknown) or its audit record unwritten (audit.unconfirmed), both exit 3; preview with the same --before to see what is still due.

A cutoff later than the current time is refused before the runtime file is read, and again against the database’s clock:

error[retention.future-cutoff] --before: the cutoff is in the future; retention erases only what has already expired
  next: Pass a --before instant that is not in the future.

Read the audit journal

Each process writes its own JSON Lines audit stream. Envelopes carry schema, eventId, time, phase, correlation, and record. Request records are accepted before protected effects; response records describe established outcomes. A dropped live request can produce unfinished. Process or destination loss can leave unmatched requests. There is no total order across streams.

File mode uses audit.path, audit.rotateBytes, and audit.retainDays. Accepted writes include fsync; stdout mode is best-effort. Applied messagingctl operations use a sibling file or stderr, keeping command stdout available for JSON results. A failed writer stays unhealthy until restart, and /ready fails. If the response append failed after a database commit, the effect still stands: inspect the operation’s stored result or use its documented retry contract, never assume rollback.

Old hash-chained files are refused by the new writer. Preserve them in a separate archive and use a fresh destination. Local segment expiry is not proof of off-host completeness: arrange shipping before retainDays removes sealed segments.

Event	Recorded when
messaging.runtime.started	The runtime starts, with the package digest, the runtime version, and the retention in force
messaging.message.accepted, .refused, .replayed	A submission is accepted, refused, or answered again from its idempotency key
messaging.attempt.started, messaging.attempt.finished	The worker starts and finishes a send attempt, with its outcome
messaging.message.quarantined	A send may have happened and the message is held as unknown
messaging.dispatch.transition	The dispatch state changes, naming whether the worker, a caller, or the operator tool changed it
messaging.message.settled	An operator settles an unknown outcome
messaging.receipt.recorded	A verified provider callback moves a delivery report
messaging.template.previewed	A sender previews a template

A record names a caller only by a keyed pseudonym of the token’s issuer and subject, and a recipient only by a keyed pseudonym. The journal never carries a subject, a contact, message content, or template data. Use a directory owned by the process user and not writable by other users or groups; audit files remain owner-only.

Watch metrics

With metricsListener set, GET /metrics on that socket answers Prometheus text. The public listener answers the same path 404 request.not-found.

Series	Labels	Counts
messaging_http_requests_total	route, method, status	Answered HTTP requests, by route template and status class
messaging_authentication_refusals_total	reason: refused, claims, profile, unavailable	Bearer credentials refused before a caller was resolved
messaging_provider_attempts_total	outcome: accepted, transient, permanent, maybe-sent	Provider attempts, by the class the attempt history records
messaging_provider_callbacks_total	outcome: unverified, unreadable, ignored, applied, unchanged, unmatched, ambiguous, unavailable	Provider callbacks, by what became of them
messaging_limit_refusals_total	limit: rate, daily, pacing, callback	Submissions an access profile’s rate or daily limit refused, attempts whose provider send slot did not open in time, and provider callbacks the callback rate refused before verification
messaging_retention_runs_total	outcome: erased, idle, failed	Retention sweeps this process ran
messaging_dispatch_jobs	state: pending, leased, unknown	Gauge read from the database at each scrape: messages waiting to send, being sent, or held for an operator

Counters are per process and start at zero at each start. The queue depth is pending plus leased; a rising unknown is messages waiting for you to settle an unknown outcome. When the database cannot answer, that scrape has no messaging_dispatch_jobs samples and the runtime logs a warning, so the series goes stale rather than reporting an empty queue. A messagingctl retention erase-expired run is a separate process: the journal records it, these counters do not.

A rising unverified callback count usually means a callback secret or URL differs between the provider and the runtime file.

Capacity

No throughput figure is part of the contract. What bounds dispatch is the provider: each runtime process sends through eight worker lanes, each claiming, sending, and recording one message at a time, so a provider that takes 200 ms to answer caps one process at 40 sends per second. A provider’s ratePerSecond and concurrencyLimit lower that further.

The product’s own measurement, with the messagingctl dev mock gateway answering every SMS after 200 ms and PostgreSQL in a container on the same Apple M5 Max running macOS, reached a steady 21.3 to 26.3 SMS per second on one process over three runs. It holds for that machine only. Run products/messaging/scripts/measure-throughput.sh from a Registry Stack checkout on a host like the one you will deploy before you size anything.

Troubleshooting

Each of these is the process’s whole output on standard error, and each exits non-zero:

Output	Cause
messaging: MESSAGING_LOG must be one of error, warn, or info	MESSAGING_LOG holds another value. Exits 2
messaging: the Messaging runtime database configuration failed: the Messaging database secret could not be resolved: the secret reference secret:file/runtime-database-url could not be resolved: no readable secret of that name exists under the configured provider	The secret file is missing or unreadable. messagingctl plan, apply, and status report the applicable database connection the same way
messaging: the OIDC issuer could not be initialized	The issuer’s discovery document or keys could not be fetched. Check reachability, or declare a static jwksSource
messaging: the Messaging schema readiness check failed: the Messaging query failed: db error	serve ran against a database messagingctl apply has not prepared
messaging: the Messaging package ledger names no active package; record package sha256:… with messagingctl apply before serving	No package was recorded. Review messagingctl plan, then run messagingctl apply
messaging: the Messaging package on disk (…) is not the package the ledger names active (…); record it with messagingctl apply, then restart	The package changed since it was recorded
messaging: package.expectedDigest is …, but the package under package.root has digest …	The package directory is not the one the runtime file pins

Next

• Registry Messaging API for the contract your callers use, including provider callbacks and every problem code.
• Author a Messaging package for the package this runtime serves.
• Registry Messaging overview for what stays with the caller’s source of record.