Harden a production deployment

View as Markdown

Registry Relay and Registry Notary enforce authentication, authorization, key publication, and audit as software controls (see Security overview). RS-SEC-G Section 9 draws a line at the deployment: secret and key provisioning, key custody and rotation, audit retention and storage, tenant isolation, transport termination and certificate management, edge rate limiting, deployment configuration, and incident response stay with the operator. This page turns that boundary into a checklist you work through before you call a deployment production-ready.

When to use this

Use this checklist before a production or evidence_grade deployment goes live, and again after any change to signing keys, audit sinks, network placement, or the declared deployment profile. It assumes you have already configured your service per the Registry Relay configuration reference or the Registry Notary operator configuration reference.

Before you start

Decide the deployment profile (local, hosted_lab, production, or evidence_grade) you are claiming; see the deployment profile section of the Registry Relay configuration reference. The profile is never inferred from hostname or network position; you declare it.
Have registry-notary doctor (Notary) or your Relay startup logs available to check gate findings as you work through each item.

Keys and custody

Configure signing keys through the provider abstraction (local_jwk_env, file_watch, or pkcs11) under evidence.signing_keys, not as inline YAML values. Startup fails closed if an active signing provider cannot be constructed. See the signing key provider reference.
Provision secrets (API-key fingerprints, Relay source credentials, Notary workload tokens, private JWKs, and the audit hash secret) only through the environment variables named in config. For local development the binary accepts --env-file; for shared environments use the platform secret store and do not check dotenv files into the repository.
Rotate a signing key by adding a new kid, moving credential profiles to it, then moving the old key to publish_only until its configured publication window ends or verifiers no longer need it. Do not delete a key verifiers may still need.
Keep RS256 scoped to the eSignet relying-party (RP) client assertion key (eSignet, the MOSIP-project identity provider, is the one integration that needs RS256). Credential profile, access token, and federation response signing use Ed25519 EdDSA or ES256/P-256; RS256 is rejected there.
If a signing key uses provider: pkcs11, confirm the deployed binary was built with the pkcs11 capability (registry-notary build-info) and validate the module, token, and key lookup with registry-notary doctor --config <path>.
Relay: confirm every env-backed fingerprint.name referenced in config exists in the runtime environment, and that no raw key, fingerprint, private JWK, or full environment dump reaches a log line; see the production checklist in the Registry Relay configuration reference.

Audit sink and retention

Configure a durable audit sink (file or syslog, not bare stdout) before you declare production or evidence_grade. A missing durable sink trips relay.audit.sink_missing / notary.audit.sink_missing, which fails startup at evidence_grade (startup_fail for both services, and already startup_fail at production for Notary).
Set audit.hash_secret_env to an environment variable holding at least 32 bytes of deployment-specific random secret material. Relay startup fails closed if it is missing, empty, unset, or weak.
Keep audit.write_policy: fail_closed (the default) unless an explicit availability exception accepts best-effort audit durability under availability_first. fail_closed means a request whose audit record cannot be written returns 503 audit.write_failed instead of a success (REQ-SEC-G-009).
If you rotate the audit hash secret, retain the old secret under your audit retention controls for any period during which older records must remain comparable to new ones, or accept that new records will not match old audit handles.
Hash-chained envelopes (prev_hash, record_hash) detect ordering gaps and accidental corruption inside the retained set, but they do not prove that every historical audit record remains present. Use off-host audit shipping when completeness matters.
For evidence_grade, configure deployment.evidence.audit_ack_cursor_path for every shipping target, including stdout and syslog. The off-host shipper writes registry.audit.ack_cursor.v1 after a successful hand-off. Registry Relay and Registry Notary report ready only while acked_at is fresh and last_acked_hash equals the live keyed audit chain tail. A matching cursor means the trusted shipper claims zero local backlog. The local cursor does not prove remote receipt or retention.
Run the shipper independently of application readiness. A signed configuration bundle produces a bundle-acceptance audit record before the service starts serving requests. The service reports 503 until the shipper acknowledges that record, then reports ready while the cursor stays at the current tail.
Replace the cursor atomically, keep it at or below 16 KiB, and mount it read-only for the Registry process. Keep the cursor path on local storage. Runtime reads use one 500 ms bounded worker and fail readiness when the file is missing, stale, malformed, unsafe, mismatched, or too slow.
Treat registry-relay doctor and registry-notary doctor as offline contract checks. They can validate the cursor timestamp and shape but cannot bind it to a live audit tail, so a fresh cursor remains unverified. An evidence_grade offline doctor run therefore reports the hard shipping gate until the live service performs tail binding.

Transport and edge

For the 1.0 single-node VM topology, follow Run single-node Compose behind your reverse proxy before exposing Relay or Notary beyond the host.
Terminate TLS and manage certificates at your reverse proxy or load balancer; RS-SEC-G Section 9 leaves transport termination and certificate management to the operator.
Keep the admin listener private. A publicly exposed admin surface trips relay.admin.public_exposure or notary.admin.shared_exposure, both startup_fail at evidence_grade.
Scrape Relay metrics from the private admin network with registry_relay:metrics_read. Alert when registry_relay_ingest_consecutive_refresh_failures reaches 3, or sooner when the deployment’s freshness service-level objective requires it. Relay deliberately keeps /ready at 200 while a valid last-good table remains available. Refresh health shows source load failures; it does not prove that business-domain data is current.
Set explicit cors.allowed_origins. The default CORS policy is deny by omission; add only the origins your integration actually needs.
Enable server.trust_proxy with an explicit trusted_proxies list only when a reverse proxy sits in front of Relay. Both enabled and trusted_proxies default to off/empty.
Enforce ingress rate limiting at your gateway or edge, then either declare deployment.evidence.ingress_rate_limit: true or accept the relay.ingress.rate_limit_missing finding it otherwise raises. The flag defaults to false.
Keep Notary’s Relay connection on HTTPS unless the two services share a network namespace. allow_insecure_localhost permits only an HTTP IP-loopback origin for that colocated topology; remote HTTP origins remain invalid.
Require every project-authored Relay source destination to be an exact HTTPS origin. The project validator rejects origins with user information, paths, queries, or fragments.
For Postgres sources, require sslmode=require on the connection string and use read-only database credentials; for live materialization, scope that role to SELECT only the configured table or view.
Apply Registry Platform HTTP-security response headers and outbound HTTP policy to product egress. Confirm that Relay source calls remain inside the compiler-pinned destination, method, path, and cumulative budget (REQ-SEC-G-012).

Deployment profile and posture endpoint

Declare deployment.profile explicitly. An omitted profile produces the deployment.profile_undeclared startup failure in both Relay and Notary. Neither product infers local or another profile from its hostname, listener, or network placement.
Treat evidence_grade as requiring a signed local config bundle: running it from a plain local YAML file trips relay.config.unsigned / notary.config.unsigned and the process refuses to start.
Review GET /admin/v1/posture regularly. It reports the declared profile, active findings, and active waivers, so the deployment’s actual state is inspectable rather than asserted. Read RS-OP-POSTURE before building operational tooling around its default or restricted tier.
Give every waiver a validated operator reference and a mandatory expiry date. Add a short summary only when operators need more context, and keep credentials and private keys out of both fields. startup_fail and readiness_fail gates are never waivable; a waiver naming one is rejected. Waiver references and summaries appear only in the restricted posture tier.
Use deployment.evidence.* flags (for example ingress_rate_limit, api_key_rotation) only to assert controls that live outside the process and cannot be observed by it. Each flag defaults to false.
If Notary runs active-active, declare multi_instance, install the typed Notary-owned PostgreSQL state schema, and require every replica to pass state doctor and runtime readiness. The process rejects state.storage: in_memory outside the local, single-instance profile.

Incident response

Know your private disclosure channel before you need it: see Report a vulnerability.
If credential-status is enabled, use the admin status endpoint (requires the registry_notary:admin scope) to mark a compromised credential revoked. Registry Notary defines no other revocation flow; a credential issued with credential-status disabled cannot be revoked through the API. See Known limitations and non-guarantees.
Rotate a compromised signing key using the documented rotation procedure (new kid, old key to publish_only) rather than deleting it outright, so previously issued, still-valid credentials remain verifiable through the rotation.
Preserve the audit trail for post-incident review: hash-chained, fail-closed audit records are the record a deployment reconstructs a request timeline from (REQ-SEC-G-008, REQ-SEC-G-009).

Verify

Start the service against your production config and confirm it starts (or fails closed as expected for a deliberately unmet startup_fail gate).
Query GET /admin/v1/posture and read the deployment object: confirm the profile matches what you declared, and that no unwaived finding at or above your profile’s threshold is outstanding. Validate the response as RS-OP-POSTURE before an automated check acts on it.
Send a request that should be audited and confirm a corresponding record lands in your configured sink, not only stdout.
Attempt an admin-listener request from outside the private network it is meant to be bound to, and confirm it is refused.

Troubleshooting

Symptom	Cause	Fix
Process refuses to start after declaring `production` or `evidence_grade`	A `startup_fail` gate tripped (for example missing audit sink or unsigned config)	Check the posture or startup log for the finding id, then fix the underlying condition; `startup_fail` gates cannot be waived
Startup reports `deployment.profile_undeclared`	No `deployment.profile` is set	Declare the actual profile explicitly before restarting; an omitted profile is not the same as `local`
Posture reports `deployment.waiver_expired`	A waiver’s `expires` date passed	Re-review the finding and either fix it or issue a new waiver with a new expiry