Operations runbook

View as Markdown

This runbook describes the V1 operating model for running Registry Relay in development, staging, and production-like deployments.

Deployment model

Recommended production topology:

Run one registry-relay process or container per deployment unit.
Bind the data plane on server.bind, usually 0.0.0.0:8080 in a container.
Put TLS, WAF rules, and external auth policy at the ingress or service mesh layer.
Keep source files mounted read-only.
Keep server.cache_dir writable by the Relay runtime identity. In the production container, that is distroless non-root UID/GID 65532:65532.
Prefer stdout audit records in containers and let the platform log pipeline retain, rotate, and forward them.
When server.admin_bind is enabled, expose it only on an internal address or private network policy.

Container defaults:

/etc/registry-relay/config.yaml       default config path
/var/lib/registry-relay/data          recommended source-data mount
/var/lib/registry-relay/cache         default writable cache mount when configured
/var/log/registry-relay               audit file mount for VM-style deployments

The binary exits non-zero if config parsing or validation fails, if required API-key hash environment variables are missing, or if listeners cannot bind.

flowchart TB
  Client["API clients"] --> Ingress["Ingress or service mesh<br/>TLS, WAF, external auth"]
  Mon["Monitoring network"] -. "scrape /metrics" .-> Admin
  AdminNet["Private admin network"] --> Admin

  subgraph Proc["registry-relay process or container"]
    direction TB
    Data["Data-plane listener<br/>server.bind"]
    Admin["Admin listener<br/>server.admin_bind, /metrics, posture, config, reload"]
  end

  Ingress --> Data
  Data --> Src[("Source files<br/>read-only mount")]
  Data --> Cache[("server.cache_dir<br/>writable")]
  Data -. "audit JSONL" .-> Sink["Audit sink<br/>stdout, file, or syslog"]
  Admin -. "audit JSONL" .-> Sink

Recommended production topology. TLS, WAF, and external auth sit at the ingress. The admin listener carries metrics, posture, governed config, and reload operations and is reachable only from the private admin and monitoring networks. Source files are read-only; the cache is writable; the public data-plane listener does not mount /metrics.

Production hardening checklist

Run this gate before promoting any deployment beyond local demo use. Items that are fully covered by an existing section link to it; items unique to this checklist carry a one-line note.

Network boundaries

Public data-plane listener is behind TLS at the ingress or service mesh layer. See Deployment model.
server.admin_bind is bound only to a private interface or loopback; it is not reachable through the public ingress. See Deployment model and Metrics.
/metrics, /admin/v1/posture, and reload routes are not accessible from server.bind. See Admin posture and config apply.
Rate-limiting is configured at the ingress for broad metadata discovery and aggregate endpoints.
server.trust_proxy is disabled unless the gateway sits behind documented trusted proxy CIDRs. See Configure.

Auth and key rotation

OIDC is preferred for multi-service deployments; API keys are acceptable only when a rotation and storage workflow is in place. See API-key provisioning and rotation.
Dataset scopes are granted narrowly: metadata, aggregate, rows, evidence-verification, and admin scopes are separate and not implied by one another.
scope_map is reviewed whenever IdP role names change (OIDC deployments).
Denied callers are tested for every exposed dataset and adapter before go-live.

Secrets handling

API-key hashes, audit.hash_secret_env material, OIDC client secrets, and database passwords are stored in the platform secret manager, not in YAML, image layers, shell history, crash reports, or issue trackers. See API-key provisioning and rotation and Audit sink and rotation.
Full environment dumps are disabled in diagnostic tooling.

Source data mounts

File sources are mounted read-only. See Deployment model.
Database credentials have read-only privileges.
PostgreSQL sources are bounded by configured projections, filters, and limits; table ids, column names, source paths, and query text are not exposed through metadata unless explicitly published.
server.cache_dir is writable only by the Relay service account (UID/GID 65532:65532 in the production container). See Deployment model.

Audit sink

An audit sink is configured before production use. See Audit sink and rotation.
audit.hash_secret_env is set to at least 32 bytes of deployment-specific random secret material; the relay fails closed if it is missing or weak.
Audit records are shipped off-host when completeness matters; local rotating files prove only the retained set’s internal chain. See Audit sink and rotation.
Identifier fields that need audit redaction carry sensitive: true in table or entity field config. Note: sensitive: true is audit-only; it does not hide fields from authorized responses.
Bearer tokens, raw API keys, raw query values, row bodies, VC-JWTs, and unreviewed detail text are not logged.

Metadata and Notary posture

Portable metadata is validated before deployment (just metadata-validate-profiles). See Build and release.
Runtime backend URLs, source paths, scope names, and table ids are absent from portable metadata manifests.
Scoped runtime metadata is not placed in shared public caches.
Registry Notary claim evaluation and credential issuance stay outside Relay. Relay holds no issuance signing key. Any Relay-native consultation is activated only from the complete reviewed artifact closure and a dedicated durable state plane.

Container runtime policy

Production images use the Dockerfile (Distroless cc-debian13:nonroot, UID/GID 65532:65532). Dockerfile.demo is not used as production runtime evidence.
No shell, package manager, curl, or wget dependencies are present in the production runtime stage; healthcheck uses registry-relay healthcheck.
Writable mounts (server.cache_dir, audit.sink: file path) are owned by UID/GID 65532:65532. See Deployment model.
TLS client behavior is verified after any base-image change by exercising an HTTPS OIDC JWKS/discovery path or a PostgreSQL TLS connection.
The container starts with a read-only root filesystem after writable cache, data, and audit paths are mounted explicitly; both the Relay process and registry-relay-rhai-worker execute as UID/GID 65532:65532, and readiness succeeds with the intended configuration and CA roots.

Readiness gates

Liveness (/healthz) and readiness (/ready) probes are configured in the orchestrator. See Readiness and probes.
Startup time allows for the largest XLSX/Parquet ingest before readiness is declared. See Readiness and probes.
Alerts are set on startup validation failures, source ingest failures, audit sink failures, and auth provider failures. See Metrics.
Degraded-source behavior and readiness expectations are tested in staging with production-shaped data sizes.
The exact config, binary version, feature flags, and metadata manifest are recorded for each deployment.

Pre-promotion test gate

Run the closest practical checks for the enabled feature set before promoting any image:

just fmt-check
just lint
just test-default
just test
just build
just metadata-validate-profiles

When optional adapters are enabled, run focused all-feature integration tests for those adapters before exposing them to consumers.

Build and release

Build a local release binary:

just build

Build a container image:

scripts/build-image.sh registry-relay:local

The helper verifies that the local registry-manifest build context is a clean checkout at the reviewed commit. Set REGISTRY_RELAY_ALLOW_UNPINNED_LOCAL_CONTEXTS=1 only for local development builds that will not be published.

The base image is built with the feature set in canonical-release-features.txt, currently attribute-release,crosswalk-runtime. A custom REGISTRY_RELAY_FEATURES value replaces that set, so standards-enabled release or lab images must retain the canonical features explicitly:

REGISTRY_RELAY_FEATURES=attribute-release,crosswalk-runtime,ogcapi-edr,spdci-api-standards,standards-cel-mapping \
  scripts/build-image.sh registry-relay:local-standards

Custom profiles use unique comma-separated Cargo feature names in alphabetical order and must name the complete transitive product feature closure. The Cargo build rejects any requested profile that differs from the effective compiled set, so its feature label stays consistent with /admin/v1/capabilities.

If release notes claim SP DCI, standards CEL mapping, or OGC EDR support, record the standards-enabled image tag or digest in the release evidence.

The build requires the pinned registry-platform, registry-manifest, and crosswalk source trees because Relay uses sibling path dependencies. For local builds, keep those checkouts next to this repository or set REGISTRY_PLATFORM_DIR, REGISTRY_MANIFEST_DIR, and CROSSWALK_DIR before running scripts/build-image.sh.

Before promoting an image, inspect the effective config and verify that every env-backed fingerprint.name is supplied by the runtime environment and resolves to a sha256:<64 lowercase hex chars> fingerprint. Do not bake API keys or API-key hashes into the image.

If the runtime config uses metadata.source.path, validate the manifest and runtime bindings before promotion:

just metadata-validate path/to/metadata.yaml
cargo test --test demo_configs_load

For standalone metadata publication, use just metadata-publish and publish the generated index.json as the discovery entry point. See metadata.md for the bundle layout.

For releases that claim DCAT-AP interoperability, validate an exported /metadata/dcat/bregdcat-ap with the SEMIC validator:

just validate-catalog-semic catalog=target/metadata.bregdcat-ap.jsonld

The release workflow uploads both the generated catalog and the SEMIC JSON report as artifacts. Treat dcatap.3_0_1_base as the minimum external profile; use stricter SEMIC profiles such as dcatap.3_0_1_full when the deployment is intended to satisfy the full European profile.

Configure

Set the config path with --config <path> or REGISTRY_RELAY_CONFIG. The container image defaults to:

registry-relay --config /etc/registry-relay/config.yaml

Important configuration blocks:

server.bind: public data-plane listener.
server.admin_bind: optional admin listener. Intended for metrics, posture, capabilities, and reload on a restricted network.
server.cache_dir: writable cache for normalized Parquet files and ingest state.
server.cors.allowed_origins: default deny when empty.
server.trust_proxy: only enable when the gateway is behind trusted proxies and those proxy CIDRs are configured.
auth.api_keys: key ids, hash env var names, and scopes.
consultation: optional restart-only authorized workload, PostgreSQL state plane, hash-pinned artifact closure, pseudonym material references, and source credential references.
config_trust: optional signed bundle trust anchor, bundle path, anti-rollback state, and local break-glass override path.
datasets[].source.path: local file path inside the container or host.
datasets[].refresh: mtime, interval, or manual.
audit: audit sink and JSONL options.

Startup config changes remain a rolling restart operation. With config_trust, Relay verifies a signed local Config Bundle v1 directory at boot, validates anti-rollback state, and starts from the embedded config. There is no admin config apply route and no hot apply. Dataset reload does not reload startup config.yaml.

Operating with Registry Notary

Registry Relay is the protected registry consultation service. Registry Notary owns claim evaluation, credential issuance, and attestation. With a maintained native consultation profile, Notary sends an authenticated purpose and one bounded input to Relay. Relay executes only the profile’s hash-pinned minimized source plan and returns its closed result envelope after durable completion. Relay can also publish metadata evidence offerings that point clients to Notary. Native consultation does not move claim evaluation or issuance signing keys into Relay.

For a combined Relay and Notary deployment, use OIDC rather than a Relay API key. Bind the consultation.authorized_workload audience and exact azp or client_id to the Notary service account, grant only the scope pinned by each public contract, and keep the PostgreSQL URL, pseudonym material, and source credentials behind the environment references named by the configuration. Configuration changes, artifact changes, and secret generation changes require a restart.

Keep raw tokens and signing material out of YAML. Use service environment variables such as REGISTRY_RELAY_CONFIG, REGISTRY_RELAY_BIND, REGISTRY_RELAY_LOG_FORMAT, and REGISTRY_RELAY_ENV_FILE; use secret indirection fields ending in _env for token hashes, audit secrets, signing keys, database URLs, and source credentials.

Bootstrap native consultation state

Bootstrap is an offline, idempotent deployment step. Relay does not create a database or database roles. A DBA must first provision one writable PostgreSQL 16 through 18 primary, one migration login, and four distinct bound roles:

an existing superuser login used only for the migration session;
an isolated NOLOGIN, non-superuser owner role with CREATE on the target database;
an isolated, non-superuser runtime login used by the normal Relay process;
an isolated, non-superuser audit-pseudonym maintenance login;
an isolated, non-superuser audit-pseudonym investigation-reader login.

The owner, runtime, maintenance, and reader roles must have no role memberships and no create-role, create-database, replication, row-security-bypass, or superuser privilege. The migration login is deliberately not used by the serving process. The three login roles need CONNECT on the target database; the owner needs CREATE there.

Store each connection URL in the deployment secret store. The runtime URL’s environment reference is already named by consultation.state_plane.database_url_env. Supply only the other environment reference names, the non-secret owner role, and the explicit key lifecycle on the bootstrap command line:

registry-relay consultation bootstrap-state \
  --config /etc/registry-relay/config.yaml \
  --env-file /run/secrets/registry-relay-bootstrap.env \
  --migration-database-url-env REGISTRY_RELAY_STATE_MIGRATION_URL \
  --owner-role relay_state_owner \
  --keyring-maintenance-database-url-env REGISTRY_RELAY_STATE_KEYRING_MAINTENANCE_URL \
  --keyring-reader-database-url-env REGISTRY_RELAY_STATE_KEYRING_READER_URL \
  --active-key-id epoch-1 \
  --active-write-deadline-unix-ms "$ACTIVE_WRITE_DEADLINE_UNIX_MS" \
  --audit-event-retention-ms "$AUDIT_EVENT_RETENTION_MS"

Use a deployment-reviewed future deadline and a retention interval that matches the audit policy. PostgreSQL supplies the authoritative activation time. Record the exact bootstrap inputs in deployment configuration management: rerunning the same command attests the installed schema and keyring, while a different schema identity, role binding, lock identity, key id, deadline, or retention value is refused as drift. Successful output contains only status values:

{"schema":"registry.relay.consultation-bootstrap-state.v1","state_plane":"installed_or_attested","keyring":"initialized"}

Run registry-relay doctor first, bootstrap before starting Relay, then give the serving process only the runtime database URL. The serving fence admits one active consultation process for a state plane. A replacement process performs takeover recovery after the former holder releases or loses its database session. Keep the migration, maintenance, and reader credentials outside the serving workload. The maintained DHIS2 profile has an end-to-end deployment checklist in its README.

Back up and restore native consultation state

The consultation database is Relay correctness state, not a registry source database and not Registry Notary state. Relay owns two schemas:

relay_state_private contains the durable audit chain, consultation attempts and completions, dispatch permits, quota buckets, materialization publication pointers, batch-child replay state, serving-fence state, and audit-pseudonym keyring metadata. Only the isolated owner role owns or reads these objects directly.
relay_state_api contains the attested SECURITY DEFINER functions. The runtime, keyring-maintenance, and keyring-reader logins receive only the function execution grants required for their roles.

Registry Notary owns registry_notary_private and registry_notary_api in its database. Do not point Notary at the Relay schemas, point Relay at the Notary schemas, or copy tables between the products. A deployment can use one PostgreSQL cluster, but each product retains its own database roles, schema ownership, migrations, and recovery decision.

Treat the following items as one Relay recovery set:

The immutable registry source inputs and the complete Relay ingest cache.
The complete consultation database, never a selection of state tables.
The Relay release, config, signed artifact closure, and PostgreSQL trust root.
The migration login name plus the owner, runtime, keyring-maintenance, and keyring-reader role names and bound object identifiers.
The chain-key epoch, both advisory-lock keys, initial keyring lifecycle inputs, audit HMAC material, and every retained audit-pseudonym key material.
The recovery timestamp or write-ahead-log position and the last externally acknowledged audit watermark available to the operator.
Access-controlled evidence that binds each source-input and cache artifact to the database active pointer and history, exact generation, and restricted content digest.

For a quiesced backup, remove traffic from every Notary that can call this Relay, stop the active Relay fence holder, and prevent another Relay process from acquiring the fence. Keep the source inputs and ingest cache unchanged, then capture them and the transactionally consistent database backup at one coordinated recovery point. If Relay and Notary are recovered as one service, either keep both products quiesced while every source, cache, and database backup is taken or use a platform snapshot that gives every store one coordinated recovery point. Independent live backups can leave the database pointer, retained cache generation, source input, or Notary completion out of agreement.

A custom-format pg_dump is suitable for a database dedicated to Relay and for restoring an empty database in the same PostgreSQL cluster, where the four bound role object identifiers remain unchanged. Run the dump through a service definition and password file so the migration credential does not appear in process arguments:

PGSERVICEFILE=/run/registry-relay/pg_service.conf \
PGSERVICE=registry_relay_migration \
PGPASSFILE=/run/secrets/registry-relay-migration.pgpass \
pg_dump --format=custom --no-owner --no-acl \
  --role=relay_state_owner \
  --file=relay-consultation-state.dump \
  --dbname=registry_relay

If Relay shares a database with another owner, have the database administrator back up the complete database with an appropriately privileged backup identity. Do not narrow the backup to the two Relay schemas.

For recovery into a different PostgreSQL cluster, use a physical backup or cluster-level managed snapshot that preserves the PostgreSQL role catalog and its object identifiers. Recreating roles with the same names does not recreate their object identifiers. The state-plane metadata binds those identifiers and the restore will fail attestation when they differ.

Restore into an empty, isolated, writable PostgreSQL 16 through 18 primary with no Relay or Notary traffic path. Restore the database and exact external recovery-set items, then rerun consultation bootstrap-state with the same owner, role connections, lock keys, chain-key epoch, active key, deadline, and retention inputs. installed_or_attested plus identical means the command accepted the existing catalog and keyring. A drift response is not a repair instruction. Do not drop either Relay schema or rerun bootstrap with different inputs to make the restore pass.

For audited SnapshotExact, verify the access-controlled recovery record before startup. Its source-input, ingest-cache, and Relay-database artifact hashes must bind the same active pointer and history, exact generation, and restricted content digest. Startup reconciles the database-authoritative publication with the retained cache before opening a newer source generation. A missing or mismatched retained generation keeps the dependent SnapshotExact publication unavailable; do not edit the cache or database pointer to make it pass.

Start one Relay without admitting traffic and require /healthz and /ready to return 200. Readiness attests the current catalog, role binding, keyring, fence, quota, audit, and materialization capabilities. It does not prove that a restore includes every previously acknowledged write.

Stale recovery points

A recovery point older than an acknowledged consultation can remove durable audit phases, completion and replay decisions, quota state, a materialization publication, or a keyring transition. Relay cannot infer that missing history from the restored database. Startup and readiness can therefore pass for a catalog-correct but stale restore.

Keep a potentially stale restore offline. Recover forward with write-ahead-log replay or point-in-time recovery until the recovery point covers the last acknowledged write and external audit watermark, then repeat bootstrap attestation and the no-traffic canary. If that reconciliation cannot be proved, do not resume the same consultation workload from the restore. Expiry of the 15-minute batch-child replay rows does not repair missing audit, quota, materialization, or keyring history.

Upgrade and software rollback

consultation bootstrap-state is a clean-or-attested installer for one exact compiled schema. It is not a general migration framework. Before an upgrade, quiesce writers, preserve the complete recovery set, and run the target release’s bootstrap command before admitting traffic. If the target reports capability drift, abort the upgrade and keep the current database unchanged until that release provides an explicit migration path.

When the old and target releases attest the same schema, software rollback can reuse the current database after all target processes stop. Restore the old binary, config, artifact closure, and external key material, attest with the old bootstrap command, and start without traffic first. A pre-upgrade database backup is safe for rollback only when the target release acknowledged no consultations after that backup. Once target traffic has written state, restoring the pre-upgrade database is a stale restore; keep the current database and fix forward unless a release-specific recovery procedure proves otherwise.

For side-by-side local compose stacks, keep the public host ports distinct while letting each container use its internal default listener. A common convention is Relay on host 18080 mapped to container 8080, and Notary on host 18081 mapped to its container listener. Native local runs usually use Relay 127.0.0.1:8080 and Notary 127.0.0.1:8081; align source base_url values with the network where Notary runs.

API-key provisioning and rotation

API-key config stores only:

a stable key id;
an environment variable name holding the SHA-256 fingerprint of the raw key;
the key’s scopes.

Recommended rotation procedure:

Run registry-relay generate-api-key --id <key_id>.
Store the emitted fingerprint in the deployment secret store.
For a secret-plane rotation, keep the same fingerprint.name and restart or roll Relay after the secret changes.
For a bundle-governed rotation, publish the fingerprint under a new immutable or versioned reference, ship a signed bundle that changes only that reference, and roll Relay.
Confirm the new key can call the intended lowest-privilege endpoint.
Update the consumer to use the emitted raw api_key.
Remove the old key entry or old secret after callers move.

Live keyring reload is not wired in V1. Treat key rotation as a rolling restart operation.

Never log raw keys, fingerprints, or full environment dumps. In issue reports, include only key ids and scope names.

Credential issuance migration

Relay no longer owns response credential issuance, DID hosting, credential schemas, credential contexts, or signing-key rotation. Remove provenance: and entity publicschema: blocks from Relay config, remove Relay issuance signing secrets from the runtime secret store, and remove probes for /.well-known/did.json, /schemas/{claim_type}/{version}, and /contexts/{vocab}/{version}.

Use Registry Notary for credential issuance, evidence verification, issuer metadata, and signing-key operations. Relay should only publish evidence offering metadata that lets clients discover the relevant Notary service.

Audit sink and rotation

Audit records are JSON Lines and are separate from operational logs. Operational logs go to stderr as readable text by default. Set REGISTRY_RELAY_LOG_FORMAT=json or REGISTRY_RELAY_LOG_FORMAT=jsonl when operational logs should be emitted as JSON Lines for collection or redirected files.

Current runtime behavior:

The public and admin listeners cap accepted sockets with server.max_connections, close incomplete HTTP/1 headers after server.http1_header_read_timeout, and bound request-body reads with server.request_body_timeout. Direct HTTP/2 serving uses the same finite connection cap and keepalive timeout, but production deployments that terminate HTTP/2 at a reverse proxy must set bounded proxy header/body read timeouts and per-client connection limits before forwarding to the relay.
audit.sink: stdout writes audit JSONL to stdout.
audit.sink: file writes audit JSONL to the configured path and rotates in-process by rotate.max_size_mb and rotate.max_files.
audit.sink: syslog ships audit JSONL to the local syslog Unix datagram socket.
Audit output is always wrapped in registry-platform-audit envelopes with prev_hash and record_hash fields. These fields detect edits, reordering, and gaps inside the retained log set, starting from the first retained record. They do not prove that earlier records were never deleted, or protect against a writer that can rewrite the entire local sink. Use off-host audit shipping when completeness matters. audit.chain is retained for config compatibility.
HTTP request completion is logged at info with method, matched route template, request id, status, and latency. It does not log raw query strings, request bodies, auth headers, or row values.
REGISTRY_RELAY_LOG_FORMAT=json switches stderr operational logs from text to JSONL.

File sink example:

audit:
  sink: file
  format: jsonl
  hash_secret_env: REGISTRY_RELAY_AUDIT_HASH_SECRET
  path: /var/log/registry-relay/audit.jsonl
  rotate:
    max_size_mb: 100
    max_files: 14

For container deployments, stdout is still the simplest default because the platform log pipeline owns retention, rotation, access control, and SIEM forwarding. For VM deployments, use file when the gateway should own audit rotation locally, or syslog when the host forwards records to a central collector.

audit.hash_secret_env is required for runtime startup and must point to at least 32 bytes of deployment-specific random secret material. The relay fails closed when the setting is missing, empty, unset, or weak, so sensitive audit lookup hashes never silently downgrade to unkeyed SHA-256.

Audit records must not contain raw secrets or raw API keys. Mark identifier fields as sensitive: true in table or entity field config when query values should be deterministically hashed in audit rather than omitted entirely. As of v0.8, the flag is audit-only; it does not remove fields from authorized API responses.

Data-Purpose audit semantics (frozen): when the Data-Purpose header is present on an ordinary entity or feature request, its value is always recorded verbatim in the audit trail (purpose field). Header presence can be required per entity via require_purpose_header: true; a missing header returns 400 auth.purpose_required. Without an entity governed_policy, purpose values are not enforced or compared on those ordinary reads. With an entity governed_policy, governed evidence-gateway routes evaluate the configured PDP purpose allowlist and return stable pdp.* denials. Value-level allowlists remain additive opt-in configuration. Native /v1/consultations/.../execute requests instead validate the purpose against the hash-pinned public contract before dispatch and record pseudonymous durable consultation evidence.

Dataset refresh and reload

Refresh modes:

mtime: poll source file modification time and reload when it changes. The default poll interval is 60 seconds.
interval: reload unconditionally on the configured interval.
manual: reload only through an admin request.

The original source file is never modified. When an ordinary refresh fails after a successful ingest, Relay keeps serving the last-good table and /ready remains 200. Relay records the failure against that resource without advancing the last successful data-load timestamp. A resource with no successful generation remains not ready.

Audited SnapshotExact adds a separate fail-closed boundary. If durable publication advances but local table registration fails, ordinary reads keep their last-good table and global /ready remains 200 when every other readiness dependency is healthy. Relay makes the affected SnapshotExact publication slot unavailable, so every dependent consultation returns consultation.unavailable instead of using the former handle. A snapshot that exceeds its execution-time freshness bound also returns consultation.unavailable; that per-execution staleness does not change global /ready. The next table-specific refresh reconciles the exact durable generation and digest before Relay opens the source for a newer generation.

For project-authored audited SnapshotExact, retain_generations accepts 1 through 16 and includes the active completed cache generation. The setting defines a bounded recovery set for cache cleanup and readers. It does not provide an operation that selects an arbitrary retained generation for rollback. Ordinary resources keep the built-in current and previous cache generations.

This refresh-health signal reports whether Relay can still poll and load a source. It does not establish semantic or domain freshness. A successful refresh can load source data whose business dates are old, while an unchanged mtime poll can prove that the source is reachable without proving that the source owner published data on time.

Manual table reload:

curl -X POST -H "Authorization: Bearer $ADMIN_API_KEY" \
  http://127.0.0.1:8081/admin/v1/datasets/social_registry/tables/individuals_table/reload

Manual reload-all for deployments without audited SnapshotExact:

curl -X POST -H "Authorization: Bearer $ADMIN_API_KEY" \
  http://127.0.0.1:8081/admin/v1/reload

The reload-all response includes status and aggregate counts for total, succeeded, and failed resources. When any source is bound to the audited SnapshotExact materialization coordinator, Relay rejects the whole reload-all request before source access, counts every configured resource as failed, returns HTTP 500 with ingest.materialization_failed, and refreshes none. Use the table-specific endpoint for every intended source. Current Relay does not provide atomic multi-materialization refresh.

When no audited SnapshotExact source is configured, reload-all prepares every configured source resource before publishing any of them. If any resource cannot prepare, Relay keeps the previous coherent ordinary-table generation active and returns HTTP 500 with status: "failed". Inspect the audit and operational logs for the resource-level failure context. This route reloads configured source resources, not startup runtime config.

Admin posture and config bundles

Admin capabilities and operations posture are read-only admin-listener routes with their own scope:

curl -H "Authorization: Bearer $OPS_READ_API_KEY" \
  http://127.0.0.1:8081/admin/v1/capabilities

curl -H "Authorization: Bearer $OPS_READ_API_KEY" \
  http://127.0.0.1:8081/admin/v1/posture

Use ?tier=restricted only for trusted operations users who need the restricted projection. The default projection is redacted for broader operational sharing. The restricted Relay posture includes per-resource relay.refresh_health observations. Each entry reports the last successful data-load time, consecutive refresh failures, and whether Relay is serving last-good data. The default tier omits these resource identifiers and observations.

The independent registry_relay:admin scope still protects reload operations:

POST /admin/v1/reload
POST /admin/v1/datasets/{dataset_id}/tables/{table_id}/reload

There are no Relay admin routes for config verify, dry-run, or apply. Signed config bundles are local boot artifacts. Verify a candidate bundle before promotion with registry-relay config verify-bundle, place the accepted bundle and trust anchor on the node, then restart Relay. Startup verification emits the acceptance audit event before anti-rollback state is advanced.

Break-glass is file-based and evaluated only during boot. A rollback override may accept the exact signed bundle hash named by the root-owned override file. An accept_unsigned override may pin an absolute local config path and hash for emergency startup; it does not advance the signed bundle high-water mark.

Governed config CLI

The registry-relay config command group verifies boot-time signed config bundles from the command line.

registry-relay config verify-bundle <flags>

config verify-bundle

verify-bundle verifies the local bundle directory, checks anti-rollback state read-only, validates the embedded Relay config, and prints a registry.platform.config_apply_report.v1 JSON report to stdout. It never persists state and never contacts a running gateway.

Flags:

--bundle-dir: local config bundle directory. Required.
--anchor-path: local trust anchor JSON path. Required.
--state-path: anti-rollback state path. Required.

The result vocabulary is verified, rejected_signature, rejected_binding, rejected_validation, rejected_rollback, and internal_error.

Example:

registry-relay config verify-bundle \
  --bundle-dir /etc/registry-relay/config/bundle \
  --anchor-path /etc/registry-relay/config/trust-anchor.json \
  --state-path /var/lib/registry-relay/config-state/antirollback.json

Readiness and probes

Use:

GET /healthz
GET /ready

/healthz is liveness only and does not check datasets. /ready returns 200 only when configured resources have ingested successfully once the readiness watch is installed. On ingest failures it returns 503 application/problem+json with failed or not-ready resources.

In orchestrators:

Use /healthz for liveness.
Use /ready for readiness and traffic gating.
Give startup enough time for the largest XLSX/Parquet ingest.

Metrics

When server.admin_bind is configured, the admin listener exposes:

GET /metrics

The response is Prometheus-style text/plain suitable for scraping from the private admin network. The public data-plane listener does not mount /metrics.

Metrics are intentionally bounded. Request metrics use low-cardinality labels such as method, route or endpoint class, and status, plus request-duration buckets. Readiness metrics are gauges derived from the ingest readiness snapshot. Metrics must not include raw query values, raw bearer tokens, request ids, API-key ids, key fingerprints, Data-Purpose values, or dataset row content.

Relay exports these private per-resource refresh-health gauges on the same protected metrics route:

registry_relay_ingest_consecutive_refresh_failures{dataset_id,resource_id}
registry_relay_ingest_last_successful_refresh_timestamp_seconds{dataset_id,resource_id}

Alert when registry_relay_ingest_consecutive_refresh_failures reaches 3. Use a lower threshold when the deployment’s freshness service-level objective cannot tolerate three failed attempts. A successful data load resets the count and advances the timestamp. A successful unchanged mtime poll resets the count without advancing the timestamp.

Troubleshooting

Startup diagnostics and generated input ownership

Relay startup failures cross the process boundary as one product-owned relay.startup.* code with static meaning and remediation. The terminal failure line remains visible when RUST_LOG=off. Listener codes identify the owning field and a closed bind-failure category without printing the configured address, port, service identity, or operating-system error:

relay.startup.data_listener_address_in_use, relay.startup.data_listener_permission_denied, and relay.startup.data_listener_unavailable refer to server.bind.
relay.startup.admin_listener_address_in_use, relay.startup.admin_listener_permission_denied, and relay.startup.admin_listener_unavailable refer to server.admin_bind.

Configuration loading is classified by the safe phase that failed:

relay.startup.config_source_unavailable covers an unreadable --config source or configured split metadata source.
relay.startup.config_environment_binding_rejected covers unsafe or unresolved environment expansion.
relay.startup.config_deprecated_field_rejected distinguishes a field that the current runtime no longer accepts.
relay.startup.config_document_invalid covers document syntax, unknown fields, and typed-schema mismatches.
relay.startup.config_validation_rejected covers cross-field and runtime binding invariants after the document is typed.
relay.startup.bundle_* and relay.startup.consultation_artifacts_rejected retain the governed bundle and consultation-closure boundaries.

Relay deliberately has no raw or verbose startup-error escape hatch. Parser, validator, source, and operating-system errors can contain country-local paths, URLs, environment names, identifiers, supplied values, or secret-bearing excerpts. Those values must not reach stderr, collected logs, CI output, or support bundles. This is the concrete confidentiality invariant behind suppressed source diagnostics. These terminal categories do not claim to emit a field path. Run authored-project validation for field-addressed guidance.

When Relay input was generated from a Registry Stack project, do not edit the generated runtime file to clear a startup error. Validate the human-authored project, correct the reported field or environment binding there, and regenerate the complete product input:

registryctl check --project-dir registry-project --environment local --explain
registryctl build --project-dir registry-project --environment local

For a directly authored Relay configuration, make the equivalent correction in its owning source and run the normal validation and review gate before restarting.

Config fails at startup:

Check YAML shape against config/example.yaml.
Confirm every env-backed fingerprint.name variable is set.
Confirm each referenced fingerprint value is a sha256:<64 lowercase hex chars> fingerprint. For API keys, regenerate a raw key and fingerprint with registry-relay generate-api-key --id <key_id>, then store the emitted fingerprint under the configured reference.
Confirm ids are lower-snake and unique.
Check vocabulary prefixes used by concept_uri and conforms_to.
For metadata.manifest.* errors, validate the portable metadata manifest.
For runtime.binding.* errors, compare runtime dataset, entity, field, filter, and relationship ids with the compiled metadata manifest.

Protected endpoint returns 401:

Confirm the request has Authorization: Bearer <key> or x-api-key.
Confirm the raw key hashes to one configured fingerprint.
Confirm the process was restarted after key changes.

Protected endpoint returns 403:

Confirm the key has the exact scope named by the entity access block.
Remember that metadata, aggregate, rows, evidence verification, and admin scopes do not imply one another.
For row or OGC feature endpoints on entities with require_purpose_header: true, include Data-Purpose.

Dataset or entity returns unknown-resource errors:

Confirm the public path uses the entity name, not the backing table id.
Confirm entity relationships target entities in the same dataset.
Confirm field filters use exposed entity field names, not hidden storage columns.

Readiness is 503:

Inspect stderr operational logs for ingest errors.
Check the source file exists at the path visible to the container or process.
For XLSX, ensure the configured sheet is a clean rectangular table. Use header_row and data_range when the file has surrounding notes.
Confirm strict schema fields match the source columns and types.
Confirm server.cache_dir is writable.

Audit records missing:

In containers, check stdout, not stderr.
Confirm audit.include_health if expecting /healthz records. /ready is always excluded so its zero-backlog shipping check cannot invalidate itself.
For audit.sink: file, confirm the parent directory exists or can be created by the Relay runtime identity. In the production container, that is UID/GID 65532:65532.
For audit.sink: syslog, confirm the host exposes the expected Unix datagram socket (/var/run/syslog on macOS, /dev/log on other Unix platforms).

Admin reload fails:

Confirm server.admin_bind is configured and reachable only from the private admin network.
Confirm the key has the independent registry_relay:admin scope.
When audited SnapshotExact is configured, do not retry reload-all. Use the table-specific endpoint for the intended resource.
Without audited SnapshotExact, use the reload-all counts plus protected operational and audit logs to identify the failed resource, then retry only that resource through the table-specific endpoint.

Metrics missing:

Confirm you are scraping the admin listener, not server.bind.
Confirm server.admin_bind is configured and reachable from the monitoring network.
Confirm the scrape credential has registry_relay:metrics_read.
Expect /metrics on the public listener to be unavailable. Depending on the auth stack, the response may be 401 rather than 404.