Inspect and diagnose

View as Markdown

Start with the product’s own check, health, readiness, and audit surfaces. Keep diagnostic output bounded and value-free.

Prerequisites

Use an active rstack installation that includes the managed Evidence component for these rstack evidence commands.
Record the binary version, package or configuration revision, and trace identifier.
Use a synthetic subject or non-production registry record.
Keep tokens, credentials, source responses, and private keys out of tickets and logs.

Inspect the runtime surfaces

Surface	Use	Boundary
`GET /health`	Process liveness	Does not check dependencies or data freshness; Registry Evidence stays healthy when a source credential is unavailable
`GET /ready`	Traffic admission	Evidence checks signing, audit, subject binding, and source credentials; it does not request evidence data
Product audit records	Security and request evidence	Local entries are not chained or signed; use copies shipped to append-only storage and join request phases by `correlation`
`GET /openapi.json`	Released API shape	Evidence serves the release artifact; it does not describe deployment-specific grants or source behavior
Evidence `GET /metrics`	Bounded operational counts and latency	Exists only on a configured separate listener and uses closed labels without request content

Query health and readiness through the same network path used for traffic admission:

rstack evidence api /health --url 'https://<evidence-host>'
rstack evidence api /ready --url 'https://<evidence-host>'

Registry Evidence’s closed problem-code catalog is the Evidence problems reference. It needs no separate CLI catalog because the codes are fixed by the public contract.

Diagnose the failure class

Confirm the process loaded the expected configuration and package.
Check health, then readiness. A healthy process may still refuse requests when a required source, audit sink, or key is unavailable.
Match the HTTP status and stable problem code to the product reference.
Use the trace identifier to locate the value-free audit or operational record.
Reproduce the request with the smallest supported client call.
Test the external dependency separately using its operator-owned diagnostic path.

Source access and authorization

Evidence collapses an unreachable source, timeout, wrong credential, wrong media type, oversized response, malformed JSON, and source error envelope into the single source.unavailable public problem. Confirm the fixed destination, trust profile, credential, timeout, and response limit from trusted local configuration. Do not widen a source binding to clear the failure.

Authorization denials similarly disclose no internal decision. evidence.denied does not say whether audience, grant, authority, purpose, subject role, or response-format permission failed. Use the value-free refusal phase in audit and inspect those inputs locally.

An ambiguous source result is one of the conditions collapsed into the value-free evidence.unavailable response. Reproduce it with synthetic fixtures instead of manually selecting a production record.

Offline fixture reproduction

Ask an offline fixture run to explain which stages it reached:

evidence evaluate --runtime-config "<deployment-target>/runtime.yaml" \
  --fixture "fixtures/<cases>.yaml" --explain
evidencectl test "<editable-project>" \
  --target "<deployment-target>" --explain

The trace names prepare, acquire, extract, derive, validate, construct, sign, and expect stages and, for multi-call acquisition, each call role. --explain-format json emits the same trace as one document.

Explanation output contains shapes only: member names, counts, and identifiers, never selector, source, fact, derived, or response values. Each fixture declares values that must never appear, and the run refuses to print a trace containing one. This facility is offline only; diagnose served requests from their public problem and audit phases.

Diagnose BReg and Casework

Start with the operator checks against the runtime configuration you intend to serve:

bregctl doctor --runtime-config /etc/breg/runtime.yaml
bregctl verify --runtime-config /etc/breg/runtime.yaml
caseworkctl doctor --runtime-config /etc/registry-casework/runtime.yaml
caseworkctl status --runtime-config /etc/registry-casework/runtime.yaml

BReg doctor checks deployment dependencies; verify compares the installed package and database. Casework doctor checks configuration, secrets, database, audit readiness, and review staffing. Casework status reports the active package and database activation state. A warning about insufficient distinct approvers calls for more eligible people on the serving team, not another claim by the person who already approved.

Probe each service’s /health and /ready, then match a failed request’s stable code to the BReg API reference or Casework API reference. Preserve its trace identifier and exact version without copying tokens or subject payloads.

For a Casework source refusal, check the human’s source profile and the pinned context contract. source.version-mismatch or source.version-unavailable requires a matching BReg/Casework deployment that reports its version. review.distinct-approver-required requires another eligible reviewer. For review.query-too-broad, narrow the queue or exact subject and check the configured budgets; live-reference filtering and sorting require a complete bounded scan. Do not substitute the service status credential for the human display credential.

If a mutation response was lost or audit failed after commit, reconcile the original operation before issuing a new key. Readiness alone cannot tell you whether that mutation committed. Use BREG_LOG or CASEWORK_LOG at info, warn, or error; neither accepts a debug filter.

Common boundaries

Symptom	Boundary to inspect
Startup refusal	Package integrity, closed configuration, secret references, and listener binding
Authentication refusal	Issuer, audience, algorithm, key set, token time, and required scope
Authorization refusal	Complete entitlement or access-profile match
Source unavailable	Fixed destination, transport trust, credentials, timeout, and response limit
Audit unavailable	Durable storage, permissions, replacement detection, and capacity
Response rejected by a client	Media type, schema version, trace identifier, and product-specific problem contract

Do not enable broader source access, disable audit, accept an unknown configuration field, or log a protected response to make diagnosis easier.

Expected evidence

Keep the exact version and revision, timestamped health and readiness status, bounded check output, stable problem code, Evidence traceId or W3C traceparent, shipped value-free audit phases, synthetic canary outcome, dependency status, recovery action, and traffic-admission decision. The public trace identifier is not the internal audit operation identifier. Do not retain the assertion, access token, credential, or source values.

These surfaces prove only the checks they run. They do not prove off-host audit completeness, legal approval, source-domain correctness, backup freshness, or an untested client journey.

Recover

Restore the last reviewed configuration or package when its credentials and data contract remain valid. Otherwise, fix forward with a reviewed revision. Re-run readiness and one realistic request before returning traffic.