Released docs. You are viewing the documentation published with v0.34.0. Development docs are available at Latest.
Use this procedure to move Evidence Gateway from disposable local signing to a strict deployment. The service signs through its own workload-local Unix-socket proxy. The service process receives no Vault or OpenBao token, and the provider retains the private key.
Prerequisites
Section titled “Prerequisites”You need:
- A Vault or OpenBao Transit mount administered outside the application workload.
- A separate provider identity and Transit key for each environment.
- An auto-auth method appropriate for the deployment platform.
- A reviewed deployment target based on the ready-to-copy templates under
products/evidence/reference/deployment-targets/. - Python 3.11 or later, run through
uv, to convert each service’s public key from PEM to JWK and compute its RFC 7638 SHA-256 thumbprint. Publish the exact public projections gives the exact command for Evidence Gateway.
The commands use a Transit mount named transit. Change the mount consistently when your provider
uses another name.
Create one key per service
Section titled “Create one key per service”Run the provider administration commands through an authenticated operator session, not from the Evidence Gateway container.
Settle three things before you create the key. Decide which provider instance and mount hold this signing identity for as long as the key is in service. Confirm that the provider’s own backup and restore covers that mount, because there is no key-level export or plaintext backup to fall back on. Record the mount, the key name, the service and environment the key serves, and the operator identity authorized to rotate it, in the deployment record rather than in application configuration.
A key created with exportable=false and allow_plaintext_backup=false never releases its private
material, and the pair is one-way in the permissive direction only: a provider administrator can set
either field to true on this key later, and Vault and OpenBao then refuse to set it back to
false. Evidence Gateway accepts a Transit key only while both fields read false, so exporting the key
retires that key name permanently, for every version under it.
There is no supported way to move this private key to another key manager or to sign with it outside
the provider. The recovery path is a new key and
rotating the signing key, not export.
Create non-derived, non-exportable P-256 keys with plaintext backup disabled:
vault write transit/keys/evidence-signing \ type=ecdsa-p256 \ derived=false \ exportable=false \ allow_plaintext_backup=falseUse the equivalent bao commands for OpenBao. The
Vault Transit API and
OpenBao Transit API define the same fields used by
the runtime checks.
Read each key’s metadata and record the nonzero version that the application will pin:
vault read -format=json transit/keys/evidence-signing > "<operator-review-path>/evidence-key.json"The reviewed metadata must report type: ecdsa-p256, derived: false, exportable: false,
allow_plaintext_backup: false, and signing support. Remove the local review files after recording
the approved public projection and controls. Do not commit provider responses.
Publish the exact public projections
Section titled “Publish the exact public projections”Convert the selected version’s public PEM to an exact public JSON Web Key (JWK). Compute the RFC
7638 thumbprint from crv, kty, x, and y, then use that 43-character value as both kid and
the filename:
{ "kty": "EC", "crv": "P-256", "x": "<base64url-x-coordinate>", "y": "<base64url-y-coordinate>", "alg": "ES256", "kid": "<rfc7638-thumbprint>"}The object must contain exactly those six members and no private member. Run this conversion from
each service’s saved review metadata; it reads the pinned version’s PEM, derives x and y,
computes the RFC 7638 thumbprint, and writes the finished JWK to the named directory as
<thumbprint>.jwk.json:
pem_to_jwk() { metadata_path=$1 version=$2 out_dir=$3 uv run --with cryptography python3 - "$metadata_path" "$version" "$out_dir" <<'PY'import base64, hashlib, json, sys
from cryptography.hazmat.primitives.serialization import load_pem_public_key
metadata_path, version, out_dir = sys.argv[1:4]
with open(metadata_path) as f: metadata = json.load(f)pem = metadata["data"]["keys"][version]["public_key"].encode()numbers = load_pem_public_key(pem).public_numbers()
def b64url(n): return base64.urlsafe_b64encode(n.to_bytes(32, "big")).rstrip(b"=").decode()
x, y = b64url(numbers.x), b64url(numbers.y)canonical = json.dumps({"crv": "P-256", "kty": "EC", "x": x, "y": y}, separators=(",", ":"), sort_keys=True)kid = base64.urlsafe_b64encode(hashlib.sha256(canonical.encode()).digest()).rstrip(b"=").decode()jwk = {"kty": "EC", "crv": "P-256", "x": x, "y": y, "alg": "ES256", "kid": kid}
with open(f"{out_dir}/{kid}.jwk.json", "w") as f: json.dump(jwk, f, indent=2) f.write("\n")print(kid)PY}
pem_to_jwk "<operator-review-path>/evidence-key.json" "<nonzero-version>" "environments/<environment>/evidence/public-keys"Each command prints the thumbprint it used as the filename. The result lands in the same environment target the command publishes to:
environments/<environment>/ evidence/public-keys/<evidence-thumbprint>.jwk.jsonKeep Evidence Gateway signing, audit, subject-binding, and client keys distinct. Run the target’s check-public-key-separation.sh before review when you
start from the maintained deployment-target templates.
Configure the governed key and runtime signer
Section titled “Configure the governed key and runtime signer”Reference the Evidence Gateway public JWK from governance.yaml:
signing: format: flattened-jws-json algorithm: ES256 activePublicJwkFile: public-keys/<evidence-thumbprint>.jwk.json publishedPublicJwkFiles: [] revokedKeyIds: [] jwksPath: /.well-known/evidence/jwks.json maximumAssertionValiditySeconds: 300 verifierClockSkewSeconds: 30Bind the matching provider key and version in runtime.yaml:
signer: kind: transit unixSocketPath: /run/registry-evidence/transit-proxy.sock mount: transit keyName: evidence-signing keyVersion: <nonzero-version> timeoutMilliseconds: 2000Do not put a provider token, auto-auth credential, or private JWK in this configuration.
Isolate provider access in local proxies
Section titled “Isolate provider access in local proxies”Give each proxy identity only read access to its named key metadata and update access to its sign endpoint:
path "transit/keys/evidence-signing" { capabilities = ["read"]}
path "transit/sign/evidence-signing/sha2-256" { capabilities = ["update"] required_parameters = ["input", "key_version", "marshaling_algorithm", "prehashed"] allowed_parameters = { "input" = [] "key_version" = [<nonzero-version>] "marshaling_algorithm" = ["jws"] "prehashed" = [true] }}The parameter constraints make the service identity usable only for the pinned version and exact
JWS signing request shape. During a planned rotation,
temporarily allow the old and new numeric versions, for example "key_version" = [7, 8], then
remove the retired version after the overlap window. The provider’s min_encryption_version
provides a second retirement control.
The Evidence Gateway proxy’s core boundary is:
vault { address = "https://vault.example.com:8200" retry { num_retries = -1 }}
api_proxy { use_auto_auth_token = "force"}
listener "unix" { address = "/run/registry-evidence/transit-proxy.sock" tls_disable = true socket_mode = "0660" socket_user = "<proxy-user>" socket_group = "<evidence-group>" require_request_header = true}Keep use_auto_auth_token at "force". Under "force" the proxy replaces any token on the
incoming request with its own auto-auth token, so the proxy identity is the only identity that
reaches the provider through this socket. Under the weaker true, the proxy attaches its auto-auth
token only when the request carries none, and a token supplied by whatever reached the socket is
used instead.
Add the deployment’s reviewed auto_auth method and provider trust settings.
Configure the proxy to:
- Listen only on the service-specific Unix socket.
- Force its auto-auth token for proxied requests.
- Require the
X-Vault-Requestheader. - Set
retry.num_retriesto-1and leaveVAULT_MAX_RETRIESunset. - Set socket ownership so only the intended workload can connect.
These settings make one application signing attempt one provider signing request. The application timeout remains the outer bound. The Vault API proxy documentation describes forced auto-auth, and the Vault Agent configuration defines the retry and request-header controls. OpenBao deployments use the corresponding OpenBao Agent configuration.
Verify before routing traffic
Section titled “Verify before routing traffic”Start the proxy before checking Evidence Gateway. Run the checks from the final execution context so the commands see the same socket, public keys, paths, ownership, and secret roots as the service:
evidence check --runtime "<candidate>/runtime.yaml"Each check reads provider metadata, verifies the pinned version and custody controls, compares the provider public key with the governed JWK, and performs a sign-and-verify self-test. A mismatch, timeout, malformed response, or unavailable proxy fails the check.
Start the service only after the check passes. Route requests only after /ready reports ready. If provider access fails later, signing fails closed and readiness reports the signer
unavailable. Readiness recovers after a successful provider self-test.
Troubleshooting
Section titled “Troubleshooting”- If metadata validation fails, compare the key type, custody controls, signing support, and pinned version with the governed configuration.
- If public-key matching fails, regenerate the JWK from the pinned provider version. Do not edit
x,y, orkidby hand. - If the proxy refuses requests, check the named-key policy, forced auto-auth state, socket ownership, and required request header.
- If checks time out, inspect the workload-local proxy and provider path. Do not add application or proxy retries.