Registry stack documentation: machine-readable Markdown. Index of all pages: https://docs.registrystack.org/llms.txt Full corpus: https://docs.registrystack.org/llms-full.txt # Inspect and diagnose a running deployment > Inspect Relay, Evidence Gateway, and Registry Mint runtime health, readiness, audit, and API shape, then classify failures with stable diagnostics. Use this procedure to inspect a running Relay, Evidence Gateway, or Registry Mint instance and diagnose source or startup failure, caller authorization failure, format mismatch, cursor error, ambiguity, or audit-write failure without collecting source rows or secret values. ## Prerequisites - Use the protected operator network and a least-privilege posture or API credential. - Know the active product and instance for Relay: its runtime file path, bound source path, and package `packageRevision` digest. Know the deployment revision for Evidence Gateway and Registry Mint. - Keep public problem responses, operator diagnostics, and protected audit records in separate access classes. - Reproduce with synthetic identifiers unless separate authority permits another probe. ## Ownership and trust boundary Public problem details give callers stable, minimized classifications; Relay's `404 resource.not_found` concealment for both an unknown and an unauthorized access profile is one instance of that discipline. Relay, Evidence Gateway, and Registry Mint each write their own audit or operational records and keep their own keys separate: Relay's audit sink, Evidence Gateway's keyed JSONL audit chain (`products/evidence/OPERATOR-CONTRACT.md`, Audit and operational data), and Registry Mint's keyed JSONL token-decision chain. No deployment exposes a cross-product view: each product reports only its own health, readiness, and audit chain. Relay V2 has no separate operator-facing posture or admin surface beyond `/health`, `/ready`, and its audit sink. {/* Evidence: crates/registry-relay-v2/src/server.rs, crates/registry-relay-v2/src/api.rs, crates/registry-relay-v2/src/cursor.rs, crates/registry-relay-v2/src/startup.rs, crates/registry-relayctl/src/lib.rs, products/evidence/OPERATOR-CONTRACT.md, docs/site/src/content/docs/reference/evidence-problems.mdx, and docs/site/src/content/docs/reference/errors.mdx. */} ## Inspect the runtime surfaces Inspect each surface for its own purpose: | Surface | Use | Boundary | | --- | --- | --- | | `GET /health` (Relay, Evidence Gateway, Registry Mint) | Process liveness | Does not check all dependencies or data freshness; Evidence Gateway's `/health` answers 200 even when a source credential is missing | | `GET /ready` | Current traffic-admission readiness | Does not prove backup freshness or country approval; Relay's `/ready` ANDs audit-sink readiness, bound SQLite source readiness, and OIDC issuer readiness when an issuer is configured; Evidence Gateway checks its signing, audit, subject-binding, and source credentials, while Mint checks that clients exist, its audit writer is healthy, and its signing provider is available | | Product audit records | Security and request evidence | Retained chain integrity does not prove complete off-host receipt; use `evidence verify-audit` for Evidence Gateway and `mint verify-audit` for Mint. Relay V2 has no equivalent offline verification command; inspect the sealed segment sequence directly | | `GET /openapi.json` | Concrete API shape for the running instance | Relay's document reflects the operations compiled into the running package and needs no credential; Evidence Gateway's is the released generated artifact, unauthenticated and independent of the deployed bundle | | Evidence Gateway `GET /metrics` (only when `metricsListener` is configured) | Request-boundary counts and latency by route, method, status, and problem code | Absent unless configured; never serves evidence routes and reveals no request content | Relay V2 has no admin listener and no posture endpoint: `/health`, `/ready`, and `/openapi.json` on its one configured listener are the complete runtime status surface. Query health and readiness through the same network path used by traffic admission: ```sh curl -fsS https:///health curl -fsS https:///ready curl -fsS https:///health curl -fsS https:///ready curl -fsS https:///health curl -fsS https:///ready ``` Fetch a running Relay instance's `/openapi.json` directly and compare it against the package's reviewed compiled operations: ```sh curl -fsS https:///openapi.json ``` Evidence Gateway's `GET /openapi.json` needs no credential either, because it always serves the same released generated artifact rather than a deployment-specific document (`products/evidence/OPERATOR-CONTRACT.md`, Startup and readiness); compare it with the [Evidence Gateway API reference](../../../reference/apis/registry-evidence/). An OpenAPI match does not prove authorization, source behavior, or claim semantics. Relay V2 currently has no separate published API reference page to compare against; the running instance's own `/openapi.json` is the source of truth. ## Diagnostic tools by product Relay V2 has no machine-readable diagnostic catalog. `registryctl tooling diagnostics --catalog operator/fixture/authoring` was Relay V1 and `registryctl`-only tooling; no `relayctl` subcommand replaces it. Diagnose a Relay V2 deployment with: - `relayctl check ` and `relayctl check --production` for an authoring-time project's compiled contract, offline. - `relayctl test ` and `relayctl test --fixture ` for offline fixture reproduction. - `relayctl inspect ` for a source database's structural shape without reading row values. It reads a SQLite file directly, before an authoring project exists. - `relayctl diff ` to compare two authoring-time projects. - `GET /health`, `GET /ready`, and `relay healthcheck` for a running instance. Evidence Gateway's closed problem-code catalog is the [Evidence Gateway errors and problems reference](../../../reference/evidence-problems/) instead, and it needs no separate CLI catalog because the codes are fixed by the frozen public contract. ## Diagnose the failure class ### Source access and startup refusal Relay V2 verifies its package, source, and runtime bindings before it starts listening. `relay serve --runtime ` refuses an unsafe path (a symbolic link, or a component with the wrong owner or mode), a package whose re-derived bytes do not match what is installed at `packagePath`, a source whose SQLite schema fingerprint has drifted from what the package pinned, a missing mandatory audit input, or a runtime binding incompatible with the package, such as a protected access profile paired with `authentication.issuer: null`. Correct the mismatched package, source, or runtime file and restart. {/* TODO[evidence]: confirm whether relay serve reports a stable, documented code per startup-refusal cause, beyond the free-text startup log line; not confirmed in crates/registry-relay-v2/src/startup.rs during this review. */} Relay V2 has no source-credential concept to fail separately from startup. A snapshot source is an immutable file Relay pins by content digest; a live read-only source is read inside one read transaction per request. Neither uses an OAuth token, an API key, or any other per-source credential, so there is no source-authentication failure class distinct from the startup checks above. {/* TODO[evidence]: confirm whether a live read-only source that becomes unavailable mid-deployment (file missing, locked, or schema-drifted after startup) produces a distinct stable runtime code, or only a generic request failure; not confirmed during this review. */} Evidence Gateway collapses every source-boundary failure (unreachable, timed out, wrong credential, wrong media type, an oversized response, malformed JSON, or an error envelope) into the single `source.unavailable` problem (`crates/registry-evidence/src/runtime.rs`, `source_failure_problem`; see the [Evidence Gateway errors and problems reference](../../../reference/evidence-problems/)). Confirm which product and which boundary produced a denial before changing a credential, a source path, or an access profile. Do not widen a Relay access profile or an Evidence Gateway source binding to clear a startup or source failure. ### Caller authorization and access-profile mismatch A Relay protected access profile without a satisfying token, and a syntactically unknown access profile, both return the same concealed `404 resource.not_found`. Relay does not fall back to a less restrictive access profile. Correct the issuer configuration in `runtime.yaml` or the caller's granted scope; do not widen the compiled access profile to clear a caller's `404`. ### Format and geometry mismatch A request for a format or geometry the selected access profile does not disclose, or a request for an unsupported response format, returns `406 format.unsupported`. Select an access profile that discloses a primary geometry, or request a format the deployment supports. ### Cursor errors Relay authenticates and encrypts every pagination cursor with a fresh nonce, binding it to the source and contract revisions, operation, access profile, disclosure profile, filters, order, selected fields, and authorization context it was issued for. It refuses a cursor that is malformed, carries invalid protection, has expired past `cursor.maximumAgeSeconds` (default `300` seconds), or no longer matches the request context, including a changed filter, order, access profile, or disclosure profile. Treat a cursor error as an expected consequence of elapsed time or changed request context, not a defect, and issue a fresh first-page request. Cursors apply only to snapshot-source list operations; live read-only sources do not support pagination. ### Ambiguity Relay V2's Record operations are exact lookup or bounded list, not fuzzy matching, so Relay V2 has no separate ambiguous-outcome category the way Relay V1's consultation lane did. A request either matches a compiled operation and access profile or it does not. Evidence Gateway keeps the ambiguity discipline for its own `POST /v1/evidence` request: an ambiguous source match is one of the conditions the `evidence.unavailable` problem collapses into a single value-free response, so timing and status code do not reveal which condition occurred (`products/evidence/OPERATOR-CONTRACT.md`, Source and selector controls; see the [Evidence Gateway errors and problems reference](../../../reference/evidence-problems/)). Reproduce with synthetic fixtures rather than selecting a record manually inside either product. ### Audit write failure For every operation, including anonymous public access, Relay writes an attempt event before source access and a terminal release, unresolved, or source-failed event before returning. When the terminal write fails, Relay withholds the response rather than serve it unaudited. Restore the audit sink and confirm write access before accepting traffic again. ### Offline fixture reproduction A reproduction that fails names the contract that broke, which says a case failed but not what stage it stopped at. Ask the Evidence Gateway fixture run to explain itself: ```sh evidence --runtime runtime.yaml evaluate --fixture "bundle/fixtures/.yaml" --explain evidencectl fixtures run --project "" --explain ``` The trace records, for every case, each stage it reached (`prepare`, `acquire`, `extract`, `derive`, `validate`, `construct`, `sign`, `expect`) and the status it reached that stage at, so a case that stopped before its stated outcome says where. `--explain-format json` renders the same trace as one document (`crates/registry-evidence/src/trace.rs`). An acquisition that makes several calls records `acquire` and `extract` once per call, in the order the plan runs them, and names each call by its role in the plan (`search`, `fetch`, `member`). A chained case that stopped reports the call it reached, not the chain as one step. The trace reports shapes only: member names, counts, and identifiers, never a response, fact, derived, or selector value. Each fixture states what must never appear in its own trace, and a run whose trace holds one of those values is refused before the trace is printed, including a run that stopped on an error. The flag is offline only. A served request has no equivalent: diagnose it from its public problem and the keyed audit chain. ### Diagnostic classes that do not apply to Relay V2 Relay V1 diagnosed failures Relay V2 has no equivalent for. Do not look for these when diagnosing a Relay V2 deployment: - **Policy-decision-point codes.** Stable `pdp.*` codes such as `pdp.purpose_not_permitted`, `pdp.assurance_insufficient`, and `pdp.evidence_stale` belonged to a policy-decision-point layer Relay V2 does not have. Relay V2's only access decision is whether a request satisfies the compiled access profile, reported as the `404` and `406` classes above. - **Consultation-activation codes.** `relay.consultation.activation.source_credentials_unavailable`, `relay.consultation.activation.unsupported_plan`, and `relay.consultation.activation.artifact_registry_invalid` all belonged to Relay V1's `consultation` lane. Relay V2 has no consultation lane, no activation-time artifact registry, and no PostgreSQL-backed consultation state. - **Stale materialization.** `relay.refresh_health`, `serving_last_good`, and the staleness diagnosis built around them assumed a periodically refreshed materialization cache. Relay V2 keeps no server-side ingest cache; a snapshot source is read from its pinned immutable file and a live read-only source is read fresh inside one transaction per request, so there is nothing to go stale between requests. - **Signed-bundle rejection codes.** `relay.startup.bundle_signature_rejected`, `_binding_rejected`, `_validation_rejected`, and `_rollback_rejected` assumed a signed product bundle and an anti-rollback ratchet. Relay V2 packages are integrity-digested through `packageRevision`, not signed, and carry no anti-rollback state, so there is no signature, binding, or rollback check to reject. ## Expected evidence Retain: - Timestamped health and readiness status without response secrets. - Stable public diagnostic codes and status classes. - Relay's package `packageRevision` digest and runtime file path identifying the deployed instance, rather than a signed bundle sequence. - Evidence Gateway's `traceId` from a problem response or the W3C `traceparent` response header, and the output of `evidence verify-audit` when a signing or audit failure is suspected. The public trace is not the internal audit operation identifier. - Registry Mint's operation identifier from its local log and the output of `mint verify-audit` when a token-decision audit failure is suspected. Do not retain the assertion or access token. - Redacted audit correlation ids and the tested synthetic canary outcome. - The traffic-admission or escalation decision. ## What this proves The `404` and `406` response classes and the cursor-error categories above identify stable, documented Relay V2 failure classes with safe remediation; Relay V2's startup refusal is evidenced by its log message rather than a separate stable code catalog. Health and readiness prove only the checks each endpoint runs; Relay V2 exposes no posture surface distinct from `/health` and `/ready`. Audit records prove the retained product event chain under the configured sink and key. The instance OpenAPI proves the exposed API description returned by that instance. These surfaces do not prove off-host audit completeness, legal approval, country acceptance, source-domain correctness, backup freshness, or an untested client journey. ## Roll back or recover Keep traffic blocked while a Relay startup refusal, a capability mismatch, or a cross-product contract failure remains. Relay V2 has no anti-rollback state to violate: recovery is installing a complete prior reviewed package, source, and runtime file and following [Replace complete revisions](../../relay/#replace-complete-revisions). There is no fix-forward sequence ratchet to satisfy first. Evidence Gateway's ambiguous match and Relay V2's `404 resource.not_found` concealment are both expected minimized outcomes, not defects to recover from; fix or narrow the reviewed selector, access profile, or source data instead of selecting a record manually inside either product. ## Escalate Escalate to security when diagnostics or logs expose secret, source, path, hash, identity, or country values. Escalate to the source owner for unexplained denials, ambiguity, or freshness. Escalate to both product owners for contract or capability mismatch. Escalate to release operations for a Relay startup refusal that persists after installing a corrected package, source, and runtime file. ## Next - [Operate Relay: replace complete revisions](../../relay/#replace-complete-revisions) - [Review the error and status code reference](../../../reference/errors/) - [Evidence Gateway errors and problems reference](../../../reference/evidence-problems/)