Harden a production deployment

View as Markdown

Registry Stack runtimes enforce their own authentication, authorization, key publication, and audit boundaries as software controls (see Security overview). RS-SEC-G Section 9 draws a line at the deployment: secret and key provisioning, key custody and rotation, audit retention and storage, tenant isolation, transport termination and certificate management, edge rate limiting, deployment configuration, and incident response stay with the operator. This page turns that boundary into a checklist you work through before you call a deployment production-ready.

When to use this

Use this checklist before a deployment serving real data goes live, and again after any change to signing keys, audit sinks, network placement, the governed Evidence Gateway bundle, or the sealed Relay package. It assumes you have already configured your services per Configure Relay and, for Evidence Gateway, Configure Evidence Gateway.

Before you start

Have the exact relayctl project that produced the package you intend to serve, and be able to re-run relayctl check --production and relayctl test against it. Relay declares no assurance profile and has no posture endpoint: its equivalent of a gate finding is a compile error, a startup failure, or a readiness failure, so the offline commands are where you find problems first.
Have relayctl diff runnable between the package currently deployed and the one you are about to deploy. It classifies meaning, disclosure, and security changes, and a security-class change is the signal that this checklist needs re-running rather than skimming.
Have your Relay startup logs available. RELAY_LOG accepts only off, error, warn, info, debug, or trace; any other value silently falls back to the crate’s own info filter rather than turning on a dependency’s logging, so confirm the value you set is one of the six.
Have evidence check runnable against the exact revision you intend to serve. Evidence Gateway has no posture endpoint, no profile declaration, and no admin surface: its equivalent of a gate finding is a startup or readiness failure, so the offline commands are where you find problems first.
When the deployment runs Registry Mint, have mint check and mint verify-audit runnable against the exact configuration and retained chain you intend to serve.

Keys and custody

Evidence Gateway: configure one active ES256 P-256 public JWK with a derived RFC 7638 thumbprint kid, plus only the public overlap keys needed for current assertions. evidence check refuses a revoked, malformed, or mismatched active signer.
Evidence Gateway: supply local-assurance private JWKs, HMAC keys, and source credentials through a secret:file/<name> reference resolved under secretProviders.file.root, never as a YAML value, a command argument, or an environment dump. The file provider reads only regular, non-symlink files below that root, each owned by the service identity at mode 0600, and checks owner, mode, no-follow, link count, and open-file identity. Keep the secret root itself operator-only. Strict deployments receive only a Transit Unix socket for signing.
Evidence Gateway: give the audit hash key and the subject-binding key independently generated raw key material of at least 32 bytes each. The file provider does not base64-decode them, so write raw bytes rather than an encoded string.
Evidence Gateway: rotate a signing key by first publishing its next public JWK, then changing the active public JWK and pinned Transit key version, while retaining the old public key for at least maximum assertion validity plus allowed clock skew. Do not remove a public key a stored assertion still needs. Follow Rotate Evidence Gateway signing keys.
Evidence Gateway: use a workload-local Vault or OpenBao Transit proxy over a Unix socket in production and evidence-grade deployments. Require P-256, signing enabled, non-exportable key material, plaintext backup disabled, and an exact governed public-key match. Local JWK signing is for local assurance only. See Configure Transit signing for Evidence Gateway and Registry Mint.
Relay: supply every secret through one of the two reference grammars its runtime file accepts, secret:env/<NAME> or secret:file/<name>, and never as a literal value. <NAME> is uppercase ASCII, digits, and underscores starting with a letter; <name> is a single flat lowercase filename with no directory component, so a nested or traversing path is rejected rather than resolved. No secret may exceed 64 KiB.
Relay: on Unix, a secret:file/<name> target must be a regular file owned by the running user with mode exactly 0600 and a link count of one; a hard-linked file is refused under every name. Relay refuses to start at all on a non-Unix target, because the runtime file’s own path-trust check fails closed where Unix ownership and sticky-directory semantics are unavailable, so do not treat a non-Unix host as a degraded-but-working option.
Relay: audit.integrityKeyRef and, where a cursor is required, cursor.integrityKeyRef are separate secret references. Give each independently generated random material and hold each under your own custody and retention controls.
Relay: pick the single OIDC signing algorithm the deployment actually uses. authentication.issuer.algorithms must contain exactly one of EdDSA, ES256, or RS256, and tokenTypes must be exactly ["at+jwt"]. There is no allowlist to prune and no HS* or none to exclude: anything else fails the closed runtime profile before the process serves. Evidence Gateway’s own access-token allowlist is configured per bundle from the same three, while its assertion signing is fixed ES256.
Relay: give the issuer an https discovery URL that is byte-identical to its canonical form and ends in /.well-known/openid-configuration, with no user information, query, or fragment. Relay derives the trusted issuer identifier by stripping that suffix, so a near-miss URL is a startup failure and not a silently different trust anchor.
Relay: it holds no signing key of its own, because it signs nothing. If your review process expects a signature over a Relay response or a Relay package, that expectation is not met by this product.
Registry Mint, when the deployment runs it: use the same active, published, revoked P-256 service key posture and Transit boundary as Evidence Gateway. Treat clients.directory as key material. Any file written there registers a client and the authority Registry Mint will assert for it, and a running process re-reads the directory on SIGHUP without a restart.
Registry Mint: keep private-key-jwt as the client-authentication default. When a managed standard client cannot sign assertions, generate a unique installation secret with mint client-secret generate, deliver its owner-only file outside configuration, and register only the printed fingerprint. Never share one secret across people or installations. Allow no more than the two-fingerprint overlap needed for a planned rotation.
Registry Mint: generate its audit HMAC key independently from its signing key, give it at least 32 bytes of random material, and keep it in a separate owner-only, non-symlink file. Retain the key for as long as its audit chain must remain verifiable.

Freeze the Evidence Gateway deployment project

Mount the governed bundle/ directory, runtime.yaml, and every captured artifact read-only and non-writable to the service process. Evidence Gateway reports a non-immutable-input error and refuses to start rather than serving from an input it could write to.

Freeze the project before validating or serving it, then validate the frozen bytes:

chmod -R a-w bundle && chmod 444 runtime.yaml
evidence --runtime runtime.yaml check
evidence --runtime runtime.yaml evaluate --fixture "bundle/fixtures/<cases>.yaml"

Review the complete simultaneously enabled bundle as one disclosure surface before deployment, covering threshold ladders, overlapping categories, increasingly precise regions, jurisdiction variants, coexisting revisions, and relationship combinations. The runtime rejects two enabled requirements that declare the same disclosureGuard family, but a declared family is your reviewed attestation, not a semantic classifier: it cannot tell that two differently labelled families are equivalent.
Confirm that no two authority paths cover the same requirement, purpose, and subject tuple. Startup validation does not detect that overlap, and at request time it denies rather than choosing, so it presents as an unexplained 403.
Keep responseFormats at the signed default unless a reviewed decision widens it. Unsigned output is releasable only where the immutable bundle and the one matched grant both name it, and it is transport-authenticated convenience, never later-verifiable evidence.

Freeze the Relay source and package

Mount the sealed package at packagePath read-only. Relay re-runs the compiler and the artifact generator over the sealed inputs at startup and compares the results byte for byte, so a modified package fails to start rather than serving quietly. That check proves integrity, not authenticity: packageRevision is a digest and nothing signs a Relay package, so provenance is whatever your build and distribution pipeline can prove, not something the runtime verifies.
Decide the source profile deliberately per binding. A snapshot source is pinned to the exact bytes present at startup; a live-read-only source reflects the file as other processes commit to it. There is no third option, and neither profile copies the file.
For a snapshot source, put the file on a read-only filesystem or leave it non-writable by its own mode bits. Relay requires one of the two and refuses to start otherwise. It also refuses a symlink, and refuses a -wal or -journal sidecar beside the file, because a sidecar means the bytes it hashed are not the whole story.
Treat continued immutability as your duty, not the runtime’s. Relay pins device, inode, length, mode, and both timestamps for a snapshot and re-checks them, but a process comparing two hashes cannot exclude a privileged writer that changes and restores the bytes between them. Keep the file on storage no other workload can write.
Review the contract, not the route table. Which operations exist, which reviewed views they read, which are public and which are protected, and which properties each disclosure profile returns are all decided in registry.yaml before the package is sealed. runtime.yaml cannot widen any of it, so a review that reads only the runtime file has reviewed almost nothing.

Audit sink and retention

Evidence Gateway: put auditStorage.path on storage whose append durability, permissions, capacity, backup, restore, and retention you own. Audit is fail-closed by contract: one access-attempt event must be durably accepted before every actual source stage and the disclosure-release event before the response is released, so an unwritable sink stops the service returning success rather than degrading quietly.
Evidence Gateway: run exactly one process per audit path. The sink takes an exclusive advisory lock on <auditStorage.path>.lock at startup, and a second process pointed at the same path fails to start. The supported shape is one replica per audit path, active/passive, recovered with a readiness probe and restart-on-failure rather than with a second concurrent writer.
Evidence Gateway: treat auditStorage.maximumFileBytes as a per-segment rotation threshold, not a ceiling on the chain. Rotation happens online with no operator action, nothing in the runtime deletes or compacts a sealed segment, and total disk consumption is therefore unbounded until your retention process acts.
Evidence Gateway: run evidence verify-audit over the whole audit directory during backup, restore, and incident procedures, and on whatever cadence your retention policy requires. Startup verifies only the active segment, so this pass is what proves sealed history was not tampered with.
Registry Mint: put audit.path on durable storage whose permissions, capacity, backup, restore, and retention you own. Mint synchronizes the token-release record before returning a token and fails closed when that write fails. Treat audit.maximumFileBytes as a per-segment threshold, not a total ceiling. Rotation happens online, but Mint never deletes or compacts sealed segments.
Registry Mint: run one process per audit.path. Mint verifies the retained chain and takes an exclusive lock at startup. Archive sealed segments oldest first, never rename or archive the active segment while Mint runs, and run mint verify-audit during backup, restore, incident procedures, and on the cadence your retention policy requires.
Relay: audit.sink is a path on storage whose append durability, permissions, capacity, backup, restore, and retention you own. It is a plain path, not a choice between transports, so there is no stdout mode to fall back to and nothing to configure away.
Relay: audit is fail-closed and there is no write-policy setting. A request whose audit record cannot be written returns 503 audit.unavailable instead of an answer (REQ-SEC-G-009). Do not plan capacity on the assumption that a full or unwritable disk degrades to best-effort logging; it stops the service answering.
Relay: readiness re-verifies the audit sink rather than trusting the check made at startup, so replacing the audit file under a running process revokes readiness. Treat a sudden readiness loss with a healthy source as an audit-path problem first.
Relay: give audit.integrityKeyRef deployment-specific random material and retain it for as long as the chain it keys must remain verifiable. Rotating it makes new records incomparable to old ones.
If you rotate an audit hash secret in any service, retain the old secret under your audit retention controls for any period during which older records must remain comparable to new ones, or accept that new records will not match old audit handles.
Hash-chained envelopes detect ordering gaps and accidental corruption inside the retained set, but they do not prove that every historical audit record remains present. Use off-host audit shipping when completeness matters.
Relay: there is no off-host audit shipping feature, no acknowledgement cursor, and no readiness gate that proves remote receipt. If your controls require audit to leave the host, ship the sink file with your own tooling and own the completeness argument yourself.
Relay: runtime.yaml’s cursor block is pagination-cursor integrity, not audit shipping. It carries its own integrityKeyRef and a maximumAgeSeconds between 1 and 86400, and any contract that declares a list or a search operation, or that shows more than one resource in its resource listing, requires it. Do not confuse the two when reading a configuration file.

Transport and edge

Terminate TLS and manage certificates at your reverse proxy or load balancer; RS-SEC-G Section 9 leaves transport termination and certificate management to the operator. Relay serves plain HTTP and has no TLS settings at all.
Evidence Gateway: bind the listener to loopback, a private IPv4 address, or a unique-local IPv6 address. The runtime schema admits nothing else, fixes tlsTermination at operator-controlled-upstream, and fixes trustProxyIdentityHeaders at false. A gateway may add publication, routing, and extra rate controls, but Evidence Gateway still validates its own token profile and never accepts a proxy-supplied identity header.
Evidence Gateway: leave metricsListener unset unless you need it. When set, it must bind an address distinct from the evidence listener and subject to the same private-address rule, it serves only GET /metrics, and the evidence listener never serves metrics. Reach it from your own network only: per-route request rates are operational information even though no label carries caller content.
Evidence Gateway: size rateLimits knowing they are per process and in memory. Running N replicas multiplies every configured limit by N and a restart resets each budget to full. Set the failed-selector budget deliberately, because it is the selector-enumeration defense rather than a throughput knob, and put per-client quotas in the gateway.
Relay: server.bind is the entire server block. There is no admin listener, no metrics listener, no CORS setting, and no trusted-proxy setting, so every one of those controls is your edge’s job. Bind Relay to loopback or a private address and publish it through a proxy you configure.
Relay: enforce ingress rate limiting at your gateway or edge. runtime.yaml’s optional quotas block is a per-operation in-memory token bucket inside one process, so N replicas multiply every configured limit by N and a restart resets each budget to full. Treat it as a self-protection floor, not as the deployment’s rate control. A contract that declares any lookup requires it.
Relay: set limits.requestTimeoutMilliseconds (1 to 120000) and limits.concurrentQueries (1 to 256) to values your host can actually sustain. These are the only request-shaping knobs the runtime file has.
Relay: it makes no outbound source request, so there is no source destination to pin, no source credential to scope, and no database connection string to harden. Its only outbound traffic is OIDC discovery and JWKS retrieval against the one configured issuer. If your review checklist has a source-egress section for Relay, it no longer applies.
Keep every Evidence Gateway source hop on a private network where possible, publish no unnecessary source port, and give Evidence Gateway a dedicated source credential scoped to exactly the operation the bundle performs. Source authorization and audit remain independently operated controls.
Apply Registry Platform HTTP-security response headers and outbound HTTP policy to product egress (REQ-SEC-G-012), and confirm that every Evidence Gateway source keeps hostname and fixed-origin verification on. Evidence Gateway ignores HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY and has no application-level proxy, so an ambient proxy variable will not silently redirect a source call.
Registry Mint, when the deployment runs it: put TLS in front of it. Registry Mint serves plain HTTP and expects termination it does not manage, while Evidence Gateway requires the token issuer and its key set to be HTTPS with no loopback exception. Keep accessTokens.lifetimeSeconds short inside its 60..=3600 bound, because a minted token is accepted until it expires and no service in this stack revokes one. Standard authorization is capped at 900 seconds. Apply bounded request-body, connection, and rate controls at the trusted ingress for /token, preserving Registry Mint’s generic invalid_client response instead of exposing credential diagnostics.

Declared posture, and why neither product has one

Neither maintained runtime asks you to declare an assurance level, and neither exposes a posture route. Both replace that with the same thing: the exact bytes they loaded, verified before they serve, and a startup that fails rather than degrades.

Relay: there is no deployment block, no local/hosted_lab/production/evidence_grade profile, no findings catalog, no waivers, and no GET /admin/v1/posture. If a runbook, ticket template, or review form still asks for a Relay deployment profile or a waiver expiry, it is describing the retired V1 runtime and needs updating rather than answering.
Relay: the checks that used to be profile-bound are now unconditional. A contract with any protected access requires a configured issuer; a contract that declares any lookup requires quotas; a contract that declares a list or a search operation, or that shows more than one resource in its resource listing, requires a cursor key. None of these is a warning you can accept, and none of them can be waived.
Relay: record the packageRevision you deployed, the digest of the package file, and the relayctl project revision that produced it. That triple plus the runtime file is the only description of what the process is enforcing, and the runtime file alone is not enough because it binds paths and limits and nothing about disclosure.
Relay: the metadata a deployment exposes about itself is governed, not operational. Service, resource, statistical-dataset, semantics, classification, and processing metadata each carry a declared visibility of public, operation-bound, or operator-only. Review those settings deliberately rather than accepting whatever the starter contract produced, because they decide what an unauthenticated reader can learn about the registry’s shape.
Evidence Gateway has no equivalent declaration and no posture route either. Its posture is the exact bytes it loaded: one governed bundle and one closed runtime file, each with its own content digest, mounted read-only, with no reload, merge, mutation, governed-field override, or fallback path. Record the bundle revision you deployed, because that identifier plus the digests is the only description of what the process is enforcing.

Evidence Gateway serves one operator-controlled trust domain per process. Mutually distrustful issuers or customers, or one issuer whose clients carry the same authority under different claim names, require a second deployment with its own bundle, signer, and audit boundary.

Incident response

Know your private disclosure channel before you need it: see Report a vulnerability.
Rotate a compromised Evidence Gateway signing key by disabling provider signing authority immediately, removing its public JWK, adding its thumbprint to signing.revokedKeyIds, and activating a checked replacement or leaving the service unavailable. Restart every affected issuer and consumer. Evidence Gateway defines no credential-status or lifecycle feature, so plan incident response around short assertion validity windows and service-key denylisting rather than recall.
Rotate a compromised source credential at its own source and restart the affected process. For a compromised Registry Mint client key, remove its public JWK; for a compromised client secret, remove its fingerprint when another valid fingerprint remains, or remove the complete registration when it is the last or only secret. Send SIGHUP to reload the client registry immediately. Already issued bearer tokens remain valid until expiry.
Withdraw a Relay disclosure by publishing a new package and restarting. Relay has no admin endpoint, no reload signal, and no way to narrow a compiled operation at runtime, so the only levers during an incident are stopping the process and deploying a corrected package. Plan for that: the mean time to withdraw a disclosure is your deployment pipeline’s, not the runtime’s.
Preserve the audit trail for post-incident review. For Evidence Gateway, take the whole audit directory including sealed segments and run evidence verify-audit over it before and after moving it; for Relay, hash-chained, fail-closed audit records are the record a deployment reconstructs a request timeline from (REQ-SEC-G-008, REQ-SEC-G-009). For Registry Mint, preserve the chain and matching HMAC key, then run mint verify-audit before and after moving it.
Expect a 403 from Evidence Gateway to tell you nothing. Every authorization refusal collapses to one generic evidence.denied problem, deliberately, so it is not an oracle for which check failed. Debug it from trusted local state in this order: the bearer principal is in the deployed bundle’s audience; a grant matches the requested requirement, purpose, and subject roles; the grant carries the claimed authority; and, only for an unsigned or credential-format request, both the bundle and that grant permit that format. The keyed audit chain records the refusal phase.

Verify

Start each service against your production configuration and confirm it starts, or fails closed as expected for a deliberately unmet gate.
Relay: confirm GET /ready succeeds, then confirm what it does not tell you. Readiness proves the package verified, the sources opened and are still the files they were bound to, and the audit sink is writable. It sends no query and proves nothing about whether the data underneath is current or correct.
Relay: fetch GET /openapi.json from an unauthenticated client and read it as an outsider would. It is generated from this deployment’s own compiled contract, so it is the most direct statement of which operations exist and which are public. If it lists something you did not intend to publish, the fix is in registry.yaml and a new package, not in the runtime file.
Relay: send a request to a protected operation with no token, with a token from the wrong audience, and with a valid token missing the required purpose claim. Confirm you get 401, 401, and 403 respectively, and that none of the three response bodies distinguishes which check failed beyond its stable problem code.
Evidence Gateway: confirm GET /ready succeeds, which rechecks the subject-binding key, the signing provider, the pinned audit sink, and every source credential, including a bounded OAuth token bootstrap where a source uses client credentials. Readiness sends no evidence-data request and probes no source data endpoint, so a passing probe is not proof that a source returns data.
Registry Mint: confirm GET /ready succeeds with at least one registered client, issue one synthetic token, and run mint verify-audit to confirm its token-release record joined the retained chain. A passing readiness probe does not prove off-host receipt or retention.
Send a request that should be audited and confirm a corresponding record lands in your configured sink, not only stdout.
Evidence Gateway: request an assertion and verify the stored response offline with evidence verify --jws <file> --jwks <file> --policy <file>, using expectations you retained independently rather than values read back out of the response.
Attempt to reach Relay’s listener directly from outside the private network it is meant to be bound to, and confirm your edge refuses it. Relay itself has no network allowlist to fall back on.

Troubleshooting

Symptom	Cause	Fix
Relay refuses to start and names the package	The package did not re-derive: the sealed bytes and a fresh compile of the sealed inputs disagree	Rebuild the package from the authoring project with `relayctl package` and redeploy; do not edit a package in place
Relay refuses to start and names the runtime file	The runtime file violates the closed profile: an unknown field, a bad bind address, a limit outside its bound, a secret reference that is not one of the two grammars, or an issuer that fails the exact OIDC profile	Fix the named field; there is no permissive mode and no partial start
Relay refuses to start on a contract it served before	The contract now has protected access with no configured issuer, a lookup with no quotas, or a list with no cursor key	Add the missing runtime block; none of the three is waivable
Relay refuses to start and names the source	The snapshot sits on writable storage, is a symlink, or has a `-wal` or `-journal` sidecar beside it	Place the file on a read-only filesystem or make it non-writable, resolve the symlink, and remove the sidecar by checkpointing the database before you stage it
Relay returns `503 audit.unavailable` and then fails readiness	The audit sink is unwritable, or the file it was bound to was replaced	Restore writable durable storage at `audit.sink` and restart; the failed requests returned no data
Evidence Gateway refuses to start on a non-immutable input	The bundle directory, `runtime.yaml`, or a captured artifact is writable by the service process	Re-freeze the project (`chmod -R a-w bundle && chmod 444 runtime.yaml`) and restart
A second Evidence Gateway process fails at startup with a sink-locked error	Two processes point at the same `auditStorage.path`	Run one writer per audit path; use active/passive with restart-on-failure rather than a second replica
A second Registry Mint process fails to start, or `mint verify-audit` rejects its chain	Another writer holds `audit.path`, or retained records were changed, reordered, or written with another key	Run one writer per path and restore the matching retained chain and HMAC key; do not delete audit records to make startup pass
Registry Mint returns `500 server_error` and then fails readiness	The process could not durably append a token decision and poisoned its audit writer	Stop traffic, preserve and verify the chain, restore writable durable storage, and restart Mint; the failed request did not receive a token
Evidence Gateway returns `403 evidence.denied` for a request you expected to work	Any one of the audience, grant, authority, or response-format checks failed; the response never says which	Work through the ordered checks in Incident response against local configuration, then read the refusal phase from the audit chain