Skip to content
Registry StackDocsv0.38.0

Deploy a registry

For the operator

View as Markdown

You have a project that passes bregctl check --production (Build a production candidate gets you there) and want it serving. At the end of this page a breg process serves a package against PostgreSQL, answers GET /ready, and exposes its metrics to your scraper.

This page covers the deploy phase only: from an empty PostgreSQL server to the first activated package. The sibling pages take over once the registry serves: Bind webhook receivers for the destinations the project declares, Change an active registry for successor packages, Retain, erase, and audit for history, retained Evidence uses, and the audit journal, and Move data in bulk for imports and exports.

Two roles take part. The author prepares a package from the project. The operator, holding the migration credential, activates the package and runs the server. One person may hold both roles on a pilot; the commands stay the same. A package carries no signature: the migration credential is what authorizes an activation, and the database records each one in its activation ledger, which bregctl status reads. Every command reads --help. Paths given to --runtime-config, --credentials, --output, and --package must be absolute.

Many operator paths are resolved through held directory descriptors rather than by name: data export output and checkpoint, data import state and checkpoint, test receipt output and the credentials file bregctl test reads, package inputs and outputs, history request files, the reviewed migrations tree, project migrate writes, and authoring sources, including each module’s SQL assets and planner scripts, which are read through the module directory the listing opened. bregctl resolves each component of these paths against the directory descriptor that holds it, then opens, creates, renames, and publishes through those held descriptors, so a directory replaced by a symbolic link after the path is validated cannot redirect the command. --runtime-config and --package are resolved by pathname in registry-breg instead, as is every bregctl dev path, and the local file apply --backup <binding>=<path> names, which registry-breg opens with O_NOFOLLOW on its final component only.

Name the real directory rather than a symbolic link: on macOS, /tmp, /var, /etc, and /home, among others, are reached through symbolic links at the root, so pass /private/tmp and /private/var for the descriptor-resolved paths above. The descriptor-based refusal is fail-closed: the Linux and macOS builds resolve every descriptor-based path this way, and a build for a platform that offers no equivalent kernel-enforced resolution refuses each one instead of falling back to resolution by pathname; the diagnostic code depends on the command, since each surface reports the refusal under its own code rather than one shared across all of them.

Install the binaries on the host that will serve, and on the operator host that holds the migration credential:

Terminal window
curl -fsSL https://github.com/registrystack/registry-stack/releases/latest/download/breg-install.sh | bash
breg --version
bregctl --version

The installer verifies every downloaded binary against the release SHA256SUMS before anything reaches the install directory, and installs breg and bregctl together or not at all. The installer does not verify release authenticity. Replace | bash with | less to read it before running it on a host you operate. BREG_INSTALL_DIR selects the install directory, and the default is ~/.local/bin. The latest URL follows the newest release; to pin one, run that release’s breg-<tag>-install.sh from https://github.com/registrystack/registry-stack/releases/download/<tag>/, which installs its own tag and refuses another. For a higher-assurance installation, follow the release verification procedure for the selected release, then rerun the installer with BREG_ASSET_DIR naming the verified directory. Binaries are published for linux-amd64, linux-arm64, and macos-arm64. Testing, packaging, activation, and serving must use the same release of both binaries, and the documentation archive matching that release describes them.

To run a source checkout instead, build both binaries from its root with the maintained build helper, which also sets the dynamic-library search path macOS needs, and use that one build for every step. Record the checkout revision with the candidate, because a source build’s version string alone does not identify its source:

Terminal window
. ./scripts/cargo-runtime-library-path.sh
registry_cargo_build "$PWD" --locked -p registry-breg \
--features registry-breg/runtime -p registry-bregctl --bins

Each release publishes the container image ghcr.io/registrystack/breg:<tag>. The release manifest published beside the binaries, registry-stack-<tag>-release-manifest.json, records the promoted digest of the image; pin that digest as BREG_IMAGE in Serve the container image rather than the movable tag. The release image is built on distroless nonroot for linux/amd64; see platform support for the artifact matrix. Its entrypoint is /usr/local/bin/breg, its default arguments are --runtime-config /etc/breg/runtime.yaml, and it exposes port 8080. Beside the runtime, the image carries the matching bregctl at /usr/local/bin/bregctl; the entrypoint stays breg. The image carries no shell, no writable directory, and no healthcheck subcommand. Mount a writable volume for the file audit destination; mount the runtime configuration and its package and secret files read-only under /etc/breg. The orchestrator probes GET /health or GET /healthz for liveness and GET /ready for readiness. Activation runs either on a separate operator host with the matching bregctl, or from the same image with its entrypoint overridden, as Activate from the image shows.

Base Registry Engine needs PostgreSQL 17 or newer, with TLS between the server and the database. A project with a crs84-point field also needs PostGIS. Create two login roles: a migration role that owns the schema and applies packages, and a runtime role the serving process uses. Keep the migration credential off the serving host when no activation is in progress.

Two separate roles are the split role mode, and production uses it. The serving process then cannot write the activation ledger or the registry state, and both apply and startup refuse a runtime role that could, naming the grant, ownership, membership, or trigger and the statement that removes it. Naming the same role for both is the single role mode that bregctl dev and the quickstart use: the ledger still catches a wrong package or database, but not someone who holds that one credential. Startup logs which mode it serves in, and plan and status report it as roleMode.

CREATE ROLE registry_migration LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
NOINHERIT NOBYPASSRLS PASSWORD '<migration-password>';
CREATE ROLE registry_runtime LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE
NOINHERIT NOBYPASSRLS PASSWORD '<runtime-password>';

Create two databases with the same shape: the serving database, and a schema-test database for test. The five managed schemas in the schema-test database must be empty when test starts, and test fills them without cleaning up after itself, so drop and recreate that database before every run; a run against a database that still holds the previous run’s objects refuses with schema-test database is not clean. In each database, as an administrator:

CREATE EXTENSION IF NOT EXISTS btree_gist;
REVOKE ALL ON DATABASE registry FROM PUBLIC;
GRANT CONNECT ON DATABASE registry TO registry_migration, registry_runtime;
CREATE SCHEMA registry_internal AUTHORIZATION registry_migration;
CREATE SCHEMA registry_data AUTHORIZATION registry_migration;
CREATE SCHEMA registry_source AUTHORIZATION registry_migration;
CREATE SCHEMA registry_derived AUTHORIZATION registry_migration;
CREATE SCHEMA registry_context AUTHORIZATION registry_migration;
REVOKE ALL ON SCHEMA registry_internal, registry_data, registry_source,
registry_derived, registry_context FROM PUBLIC;

Base Registry Engine creates every table, function, and policy inside those five schemas from the package. It never creates a schema, an extension, or a role, so a missing prerequisite surfaces at test or apply rather than at runtime.

For a spatial project, add a bounding-box role named after the runtime role with the suffix __spatial_bbox, and install PostGIS in a dedicated schema that neither registry role owns:

CREATE ROLE registry_runtime__spatial_bbox NOLOGIN NOSUPERUSER NOCREATEDB
NOCREATEROLE NOINHERIT NOBYPASSRLS;
GRANT registry_runtime__spatial_bbox TO registry_migration
WITH INHERIT FALSE, SET TRUE, ADMIN FALSE;
-- in each database
CREATE SCHEMA registry_spatial_ext AUTHORIZATION postgres;
CREATE EXTENSION IF NOT EXISTS postgis WITH SCHEMA registry_spatial_ext;
REVOKE CREATE ON DATABASE registry FROM PUBLIC, registry_migration, registry_runtime;
REVOKE ALL ON SCHEMA registry_spatial_ext FROM PUBLIC;
GRANT USAGE ON SCHEMA registry_spatial_ext
TO registry_migration, registry_runtime, registry_runtime__spatial_bbox;

The maintained runtime validates these ownership boundaries when it prepares spatial storage.

PostgreSQL’s defaults let it start on almost any machine; they are not sized for serving. On a dedicated database host, begin from these values and adjust them against your own measurements. shared_buffers and max_connections take effect only after a server restart.

SettingStarting pointWhy
shared_buffers25% of memory on a host with 1 GB or more; a smaller share below thatPostgreSQL’s own page cache. Past 40% it rarely helps, and a larger value usually wants a larger max_wal_size.
effective_cache_sizeshared_buffers plus the memory the operating system can spend caching PostgreSQL filesA planner estimate that allocates nothing; a higher value makes index scans more likely.
work_memThe 4 MB default, until measurements show sorts or hashes spilling to temporary filesIt applies to each sort or hash operation in each session, so total use can reach many times the value.
maintenance_work_memAbove the 64 MB default when memory allowsVACUUM and CREATE INDEX use it, and autovacuum may take up to autovacuum_max_workers times the value.
random_page_costBelow the 4.0 default, toward seq_page_cost (1.0), when the database fits in server memoryLowering it relative to seq_page_cost makes the planner prefer index scans.

Budget connections. Every breg replica opens up to database.pool.maxSize runtime connections. Operator commands such as bregctl apply, bregctl doctor, and bregctl data import open their own sessions, and so do monitoring agents and administrators. All of them must fit in the connections PostgreSQL leaves for ordinary roles: max_connections minus superuser_reserved_connections minus reserved_connections. At startup breg logs, and bregctl doctor reports, the numbers it compared under postgres.connections.budget. When one replica’s pool alone may take more than half of the usable connections, the advisory is the postgres.connections.pool_over_half warning instead. The check sees one replica; add the others yourself.

Find sessions by name. Runtime sessions report breg as their application_name and operator commands report bregctl. A name in the connection URL is kept, so ?application_name=breg-a tells replicas apart in pg_stat_activity:

SELECT application_name, state, count(*)
FROM pg_stat_activity
WHERE datname = 'registry'
GROUP BY application_name, state;

Runtime sessions also carry idle_in_transaction_session_timeout of 120 seconds as a startup option placed after any options in the URL. PostgreSQL ends a runtime session that sits idle inside an open transaction for longer, the transaction rolls back, and the pool opens a replacement. Operator command and migration sessions carry no such bound, and no session carries idle_session_timeout.

Runtime sessions also carry jit=off as a startup option placed before any options in the URL, because row-level security inflates the planner’s cost estimate for short collection pages past the JIT thresholds and compiling such a plan costs more than running it. To turn JIT back on, add -c jit=on to the URL options; operator command and migration sessions keep the server default.

Pool in session mode only. Base Registry Engine holds session-level advisory locks and sets session settings, so a pooler in front of it must hand each client one server connection until the client disconnects. Transaction pooling is not supported. PgBouncer raises an error for a startup parameter it does not track unless ignore_startup_parameters lists it, and ignores a listed one; with options in that list it ignores the unknown parameters inside options, including the idle bound and jit=off (PgBouncer configuration). When a pooler drops the startup options, set both on the runtime role, where they then also apply to operator commands that connect as that role:

ALTER ROLE registry_runtime SET idle_in_transaction_session_timeout = '120s';
ALTER ROLE registry_runtime SET jit = off;

Record statement timings. Add pg_stat_statements to shared_preload_libraries, restart PostgreSQL, and then, as an administrator in the registry database:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Base Registry Engine’s catalog checks accept the extension either way, but the postgres.pg_stat_statements.unavailable advisory only stops once the preload above actually took effect. PostgreSQL lets CREATE EXTENSION succeed without shared_preload_libraries, then refuses every query against the view, so skipping the restart leaves doctor advising, with a message naming the module as installed but not loaded. It also keeps advising while pg_stat_statements.track is none or compute_query_id is off, because the module then records nothing. Only superusers and roles with the privileges of pg_read_all_stats see the statement text of other roles.

Keep maintenance running. Leave autovacuum and track_counts on; with either off, breg and doctor warn, because dead rows accumulate and planner statistics go stale. After a large bregctl data import, run ANALYZE in the registry database as an administrator so the planner sees the new rows before autovacuum reaches them. Take a backup before every activation, as Activate the successor describes.

On a managed service, the extension lists of Amazon RDS for PostgreSQL, Azure Database for PostgreSQL flexible server, and Google Cloud SQL for PostgreSQL each include btree_gist, pg_stat_statements, and PostGIS 3.5 or newer for PostgreSQL 17 (checked 2026-09-25). Set the parameters in the table, and shared_preload_libraries, through the provider’s parameter settings; doctor reads the values the server actually runs with.

The runtime file binds one package to one database, one token issuer, and one listener, plus an optional private metrics listener. It is a deployment artifact: keep it with the operator, outside the authoring project, and never put a credential in it. Values such as secret:file/<name> resolve to owner-only files under the file provider root; secret:env/<NAME> reads an environment variable when secretProviders.environment is declared.

apiVersion: registry.registrystack.org/breg-runtime/v1alpha1
kind: BRegRuntimeConfig
listener:
bind: 127.0.0.1:8080
publicOrigin: https://registry.example.org
identity:
environment: production
instanceId: civil-registry-1
databaseId: civil-registry-db-1
databaseInitializationEnvironment: production
secretProviders:
file:
root: /etc/breg/secrets
database:
runtimeUrlRef: secret:file/runtime-database-url
migrationUrlRef: secret:file/migration-database-url
pool:
maxSize: 8
roles:
migration: registry_migration
runtime: registry_runtime
package:
root: /var/lib/breg/packages/build-1/package
expectedDigest: sha256:<digest of the package SHA256SUMS file>
authentication:
oidc:
issuer: https://issuer.example.org
audience: breg
allowedAlgorithm: ES256
accessTokenType: at+jwt
scopeClaim: scope
scopeSeparator: " "
allowedClients: [registry-console]
deniedKids: []
maxTokenLifetimeSeconds: 300
leewayMilliseconds: 30000
jwksSource:
kind: discovery
authorityClaims:
principal: registry_principal
purpose: registry_purpose
audit:
hashKeyRef: secret:file/audit-key
destination: file
path: /var/lib/breg/audit/breg.jsonl
cursor:
secretRef: secret:file/cursor-key
eventDestinations: {}

package.expectedDigest is optional. When you set it, copy the packageDigest reported by the successful bregctl package publication. Startup checks that digest before it checks that the database is the one identity.databaseId names and has activated this package. Rebuild a candidate with bregctl package; do not edit SHA256SUMS, REVISION, or files below package.root in place.

publicOrigin may include a deployment path prefix, such as https://registry.example.org/registry-a. Discovery, paging, and schema links preserve the configured prefix and never derive their authority from request headers.

SectionWhat it binds
listenerThe bind address and the optional public origin that generated documents cite. The runtime reads neither peer addresses nor forwarded headers, so client addresses and TLS termination are enforced at your proxy.
identityThe environment, instance, and database this file may serve. environment must equal databaseInitializationEnvironment. The first apply records databaseId in the database; every later plan, apply, and startup refuses a file that names another. instanceId names the source of every webhook event this runtime captures, and the delivery worker dead-letters a stored event whose source names another instance, so startup refuses a changed instanceId while pending deliveries were captured under the previous one; keep the previous instanceId until they drain, then change it.
secretProvidersThe file root (owner-only files) and, when declared, the environment provider.
databaseSecret references for the runtime and migration connection URLs, pool bounds, and the two role names the package’s policies are written for.
packageThe package directory the activation ledger names as active and, optionally, the expectedDigest it must carry.
authenticationThe OpenID Connect verifier and the names of the claims that carry the caller’s principal and purpose.
audit and cursorThe audit destination and keyed-reference secret, and the secret that signs pagination cursors. Changing the audit key changes future pseudonyms; changing the cursor secret invalidates existing cursors.
eventDestinations and eventDeliveryOne binding per webhook destination the package declares, and payload retention; see Bind webhook receivers.
evidenceProvidersOne binding per Evidence provider the package declares, keyed by provider ID: baseUrl, trustBindingId, trustedJwksRef, revokedKeyIds, optional caBundleRef, and exactly one of tokenRef or a refreshing privateKeyJwt credential. The script cannot replace these bindings; see Governed registry actions.
fieldEncryptionKey custody for encrypted fields; declare it when the project encrypts a field. The optional provider binding’s kind member selects the custodian. kind: transit requires unixSocketPath, mount, and keyName, with an optional timeoutMilliseconds (default 5000, at most 30000). kind: localFile requires dekRef, a secret:file/<name> reference to one base64 data-key file, and is for local assurance only. The model and its limits are in Field encryption for restricted fields.
attachmentStoragePostgreSQL by default, or an operator-bound S3-compatible backend for request attachments; see Attachment storage.
attachmentVerificationOptional asynchronous HTTP verifier; content stays quarantined until approved. Configure it before the first attachment upload; see External attachment verification.
operationalTimeoutsOptional HTTP request, record lock, migration lock, migration statement, and shutdown grace bounds.
metricsListenerOptional second binding, private to the operator, that serves GET /metrics; see Scrape metrics.

The fieldEncryption block is optional; its provider binding names the custodian:

fieldEncryption:
provider:
kind: transit
unixSocketPath: /var/run/vault/transit.sock
mount: transit
keyName: breg-field-dek
timeoutMilliseconds: 5000

An encrypted field moves part of the registry’s availability into key custody, so plan the Transit binding’s custody before the first encrypted field is activated:

issuer must be an https URL without credentials, query, or fragment; plain http is accepted only on an IPv4 loopback host, for local development or an issuer or proxy the operator runs on the same host. Use an https issuer in production. audience is at most 512 characters. jwksSource takes one of three kinds: discovery (the default) reads the key set location from the issuer’s discovery document, uri names it directly with a uri member under the same URL rule, and static pins a document through documentRef.

leewayMilliseconds must be a whole number of seconds and at most 300000. Static keys load at startup; rotate them through a configuration change and restart:

jwksSource:
kind: static
documentRef: secret:file/<jwks-document>

Tokens signed with a key absent from the pinned document are refused with the value-free authentication.refused. Search the server’s JSON logs for unknown or disallowed key identifier in fields.message, with logging set to warn or info. The server emits this warning at most once per minute per authenticator. This warning applies to both static and discovery sources. An arbitrary token or a deliberately denied key can cause the same warning, so the warning alone does not establish provider key rotation.

If fresh logins fail and you independently confirm that the provider rotated its keys or regenerated them after state loss, recover a static pin with these steps:

  1. Fetch the provider’s current JWKS document from its published jwks_uri over TLS, the same way the initial pinning did. Confirm the keys with the provider operator if the rotation was not announced.
  2. Prepare and review a compatible public-key subset of that document. Keep only signing keys for your configured allowedAlgorithm, with unique kid values absent from deniedKids. Preserve their public key material and identifiers. Do not relabel algorithms or turn encryption keys into signing keys. The document must contain only a keys array with 1 to 128 keys and fit within both the 65536-byte secret limit and jwksCache.maxDocumentBytes (65536 bytes by default). Each key must use the matching public-key shape and only the members kty, kid, alg, use, key_ops, crv, x, y, n, and e that apply to that shape. Remove certificate metadata such as x5c; never include private key material. If present, use must be sig and key_ops must be exactly ["verify"]. Review the resulting subset with the provider operator before use.
  3. Replace the secret that documentRef names. For secret:file/<jwks-document>, replace the file while keeping the secret-file rules. For secret:env/<NAME>, when secretProviders.environment is enabled, replace the injected environment value in your service configuration with the document bytes. Keep the value out of command arguments and logs.
  4. Restart breg; the static document is read only at startup. If startup refuses the document, recheck its members, key shapes, algorithm, denied keys, and size before retrying.
  5. Verify with a fresh provider login that authentication succeeds again, and that tokens signed by the removed keys are still refused.

If fresh logins are still refused, confirm that every serving instance restarted with the intended documentRef, then check the issuer, audience, algorithm, token type, and key policy with the provider operator. Keep verification enabled. Restore a previous pin only if its keys remain trusted and the provider confirms they are still valid; never restore a revoked key to end an outage.

If a token signed by a removed key is accepted, stop routing protected traffic to the affected instance. Check for an old process or another serving instance using the previous configuration, and escalate the trust discrepancy to the deployment and identity-provider operators. Resume traffic only after the intended pin accepts current keys and refuses removed keys.

With kind: discovery, the runtime reads the issuer’s discovery document and caches its JWKS under the optional jwksCache bounds.

Create the files the example references as the user that runs breg and the operator commands, because the resolver refuses a file owned by anyone else. Each must be a regular file with mode 0400 or 0600 and no other link or symbolic link to it, and its name starts with a lowercase letter and uses lowercase letters, digits, ., _, and -. The bytes are used exactly as written, neither trimmed nor decoded, so write them without a trailing newline:

Terminal window
install -d -m 0700 /etc/breg/secrets
(umask 077; openssl rand -hex 32 | tr -d '\n' > /etc/breg/secrets/audit-key)
(umask 077; openssl rand -hex 32 | tr -d '\n' > /etc/breg/secrets/cursor-key)
(umask 077; printf '%s' 'postgresql://registry_runtime:<runtime-password>@db.example.org:5432/registry' \
> /etc/breg/secrets/runtime-database-url)
(umask 077; printf '%s' 'postgresql://registry_migration:<migration-password>@db.example.org:5432/registry' \
> /etc/breg/secrets/migration-database-url)

The audit key and the cursor secret each need at least 32 bytes; 64 hexadecimal characters satisfy both. Percent-encode a password that carries reserved characters. The user in each URL must equal the role named under database.roles, and the runtime requires TLS on both connections itself, so the URL needs no sslmode parameter and a server without TLS refuses the connection.

Base Registry Engine verifies a bearer access token before it reads any claim. Configure the issuer so that every token for this deployment satisfies the table. Fixture claims from journeys.yaml are never accepted by a running server.

Token partRequirement
alg, typ, kid headersalg equals allowedAlgorithm; typ matches accessTokenType case-insensitively, and when accessTokenType names the RFC 9068 access-token media type (at+jwt or application/at+jwt), either spelling of that one type is accepted; kid is present in the JWKS and absent from deniedKids.
issEquals issuer.
audA string equal to audience, or an array of 1 to 16 distinct nonempty strings containing that exact resource audience.
exp, iat, nbfThe lifetime is at most maxTokenLifetimeSeconds; clock skew up to leewayMilliseconds is tolerated.
azp or client_idListed in allowedClients when that list is not empty.
scopeThe single claim named by scopeClaim covers every required permission. It is a string split on scopeSeparator, or a JSON array of permission strings. Other shapes are refused; permissions from several claims are not merged.
principal claimThe claim named by authorityClaims.principal carries the identity recorded in audit and workflow decisions. It must match each authenticated profile’s principalClaim.
purpose claimThe claim named by authorityClaims.purpose is required when the profile lists requiredPurposes.
assignment claimsEach row boundary or claim-backed lookup names its own claim. An equals boundary needs a scalar; an in boundary needs a JSON array. An identity claim does not imply district or team assignments.

Set maxTokenLifetimeSeconds to your deployment’s chosen limit, up to 7200 seconds. The limit bounds the accepted difference between issuance and expiry; increasing it extends how long an already issued token may carry old permissions. Match the issuer’s token settings and test both accepted and refused lifetimes. Use a resource audience dedicated to this registry API, distinct from your application’s login client identifier. Obtain access tokens for that resource; an ID token issued to the login client is not an API credential.

Select the principal claim explicitly. sub is supported when selected in both the runtime and profiles; the server never falls back to another identity claim if the selected claim is missing. An issuer’s sub usually belongs to that issuer’s identity namespace. For an issuer migration, prefer an institutional identifier carried in a custom claim by both issuers, and document who guarantees that its values remain unique, stable, and never reassigned. Base Registry Engine treats the selected value as an opaque identity and does not link accounts across issuers. Changing it can change whether someone is recognized as a submitter or a previous reviewer.

Use the same principal claim in an ownership row boundary when the record’s owner field stores that identifier. Use separate assignment claims for district, tenant, or team membership. Configure the issuer to derive these claims from trusted assignments rather than caller-supplied values.

Configure a compatible OAuth issuer for registered machine clients. A client-credentials flow represents a service, not an individual signing in. Human login, account lifecycle, and multifactor authentication belong to your identity provider and application. When replacing an issuer, verify the same permitted and refused requests with real tokens before changing the serving configuration. Compare issuer, resource audience, token headers, permission source, principal identity, purpose, and assignment claim shapes.

Rotate signing keys and handle issuer outages

Section titled “Rotate signing keys and handle issuer outages”

With discovery, jwksCache.cacheTtlSeconds defaults to 600 seconds. An unfamiliar kid can trigger a refresh, subject to refreshCooldownSeconds (30 seconds by default) and the bounded negative cache. Publish a new public key before issuing tokens with it, then verify a token using that key against the registry. Keep the previous public key available while its legitimate tokens remain valid. Removing a key from the issuer does not immediately remove an already cached key from the registry.

During a fetch failure, a known cached key may remain usable until its age reaches the cache TTL plus outageToleranceSeconds, which defaults to 900 seconds. The default total allowance is therefore 1500 seconds from the last successful key-set fetch, not from the start of the outage. Unknown keys cannot use this allowance, and the server cannot obtain keys for a cold start from an unavailable issuer. After the allowance, verification requiring those stale keys fails until fetching succeeds. Token expiry and every other verification rule continue to apply during the allowance.

Short token lifetimes bound how long issued permission and assignment claims remain usable. Disabling an account or changing a group at the issuer does not update claims in an already issued token. For a compromised signing key, add its identifier to deniedKids and restart every serving instance with that configuration. That rejects all tokens signed with the denied key, including otherwise legitimate tokens. Static JWKS documents also require a configuration change and restart to rotate. Rotate credentials, keys, certificates, and trust covers related rotation boundaries for Relay and Evidence Gateway.

Change the issuer without losing request context

Section titled “Change the issuer without losing request context”

The runtime trusts one configured issuer and reads its configuration at startup. Plan a coordinated restart and client token change; one process does not provide an overlap period accepting both issuers. Keep the selected principal values stable if existing ownership and review decisions should continue to recognize the same actors. Test new tokens before the serving cutover, including refused permissions and assignment shapes.

Resolve uncertain writes before changing issuer mappings, profiles, or packages. After a timeout, retry the same request with its original idempotency key and the same effective principal, profile, purpose, and row context. A replacement token can satisfy those conditions; changing identity or authority context can produce idempotency.conflict instead of a replay. The binding also covers the request and the activation that serves it, so issuing a fresh key blindly can duplicate a write whose first response was lost. Mutation retry rules explain the request contract.

Pagination cursors bind the principal, profile, purpose, row scope, query, projection, and compiled registry identity. They also expire, with cursor.maxAgeSeconds defaulting to 300 seconds. A token renewal alone does not require starting over when the effective context is unchanged, but changed mappings, policy, package, or cursor secret can invalidate an existing cursor. Restart the query from its first page when that happens; do not treat a cursor as portable authority between deployments.

test compiles the project, applies it to the empty schema-test database, measures the resulting schema, runs every journey in tests/journeys.yaml over real HTTP with real tokens, and writes a receipt that package later binds to. Prepare three inputs:

  1. A test runtime file: the same document as the serving one, with database pointing at the schema-test database and package.root an empty directory.
  2. Real access tokens from your issuer for each journey principal, stored as owner-only files under the secret root.
  3. A credentials document binding every journey step to a token or to anonymous access:
apiVersion: registry.registrystack.org/breg-schema-test-credentials/v1
kind: SchemaTestCredentials
bindings:
- journeyId: record-lifecycle
stepId: create-record
credential:
type: bearer
tokenRef: secret:file/schema-test-token
- journeyId: record-lifecycle
stepId: get-record
credential:
type: bearer
tokenRef: secret:file/schema-test-token
- journeyId: record-lifecycle
stepId: list-records
credential:
type: anonymous

Then run the test:

Terminal window
bregctl --format json test ./my-registry \
--runtime-config /srv/registry/runtime-test.yaml \
--credentials /srv/registry/schema-test-credentials.yaml \
--output /srv/registry/schema-test-receipt.json

The report names the candidate registryRevision, the measured schemaFingerprint, and the journeys that passed. Keep the receipt and the fingerprint with the candidate: package refuses a receipt taken for different sources or a different baseline. A journey failure reports the journey and step without record values.

package seals and publishes the deployable package directory in one step:

Terminal window
bregctl --format json package ./my-registry \
--test-receipt /srv/registry/schema-test-receipt.json \
--output /srv/registry/build-1

The schema fingerprint comes from the test receipt; --schema-fingerprint states it explicitly when you want the command to fail on any other value. The report’s packageDigest names the package, and the package directory is build-1/package. The command refuses an output directory that already holds a published package, so each candidate needs a build directory of its own.

Write the serving runtime file with package.root pointing at build-1/package and, if you pin it, package.expectedDigest equal to the reported packageDigest. Rehearse the activation with the migration credential, then activate:

Terminal window
bregctl --format json plan \
--runtime-config /etc/breg/runtime.yaml \
--package /srv/registry/build-1/package
bregctl --format json apply \
--runtime-config /etc/breg/runtime.yaml \
--package /srv/registry/build-1/package --initial

plan makes the checks apply makes before it changes anything, under the same lock, and rolls all of them back: it writes nothing, appends no audit entry, and exits 0 with pending: true and activation: initial when apply would succeed, or 1 with the refusal apply would give. --initial activates the first package in an uninitialized database, and apply reports the activationId of the row it recorded in the activation ledger. bregctl status reads that ledger back without taking the lock:

Terminal window
bregctl --format json status --runtime-config /etc/breg/runtime.yaml

It reports the databaseId, the activePackageDigest with its activationId and registryRevision, the roleMode, the schema fingerprint and maintenance state, and one ledger entry per activation, each with its outcome. Confirm the configuration before starting the process:

Terminal window
bregctl verify --runtime-config /etc/breg/runtime.yaml
bregctl doctor --runtime-config /etc/breg/runtime.yaml
breg --runtime-config /etc/breg/runtime.yaml

verify opens no runtime dependency. It verifies the configured package’s files against its sum file and, when set, package.expectedDigest, re-derives every generated artifact from the package’s own source, and reports the package digest, the registry id, version, and revision plus inventory counts. It does not say whether the database is on that package; status and readiness do, and startup refuses to serve a package the ledger does not name as active. doctor opens every startup dependency without binding the listener: the runtime file, the package, the database connection and its readiness, the audit key and destination writability, the cursor key, the OIDC verifier’s key material, the event destination bindings, the retained review authority and executor bindings, the authentication profile’s claim mapping, accepted algorithms, and audience against the package this runtime serves, and the field-encryption provider and its stored key. It names the first dependency that refuses and stops there. A run that reaches the end lists every dependency it checked, then the PostgreSQL advisories from Set a PostgreSQL baseline. Advisories never change the exit status. This run is against a local bregctl dev session, whose runtime file sets database.pool.maxSize: 4 and whose PostgreSQL keeps its default connection settings and has no pg_stat_statements extension, so it also reports the statement-timings advisory:

10 dependency checks passed.
runtimeConfig pass
package pass
database pass
audit pass
cursor pass
authentication.oidc pass
eventDestinations pass
reviewBindings pass
authentication pass
fieldEncryption pass
PostgreSQL advisories:
information postgres.connections.budget
one replica's runtime pool takes at most half of the connections PostgreSQL leaves
for ordinary roles
poolMaxSize 4
maxConnections 100
superuserReservedConnections 3
reservedConnections 0
usableConnections 97
information postgres.pg_stat_statements.unavailable
pg_stat_statements is not installed in this database, so per-statement timings are
unavailable when diagnosing load
{
"ok": true,
"command": "doctor",
"checked": [
"runtimeConfig",
"package",
"database",
"audit",
"cursor",
"authentication.oidc",
"eventDestinations",
"reviewBindings",
"authentication",
"fieldEncryption"
],
"advisories": [
{
"code": "postgres.connections.budget",
"severity": "information",
"message": "one replica's runtime pool takes at most half of the connections PostgreSQL leaves for ordinary roles",
"observed": {
"poolMaxSize": 4,
"maxConnections": 100,
"superuserReservedConnections": 3,
"reservedConnections": 0,
"usableConnections": 97
}
},
{
"code": "postgres.pg_stat_statements.unavailable",
"severity": "information",
"message": "pg_stat_statements is not installed in this database, so per-statement timings are unavailable when diagnosing load",
"observed": {}
}
]
}

Start the process and check GET /ready before admitting traffic; GET /health reports liveness only.

Restarting this same version with the same runtime file, active package, and database resumes the durable review submission, result reconciliation, and automatic application jobs. A restart does not recreate a Casework review and does not bypass current Base Registry Engine application authority.

doctor and readiness report setup and dependency health. They are not a change-request workflow console. To trace one review, read the request under a profile granted readableRequestFields: [review_state] and inspect data.request.review. Its closed sections are submission, result, delivery, application, and recovery; identifiers and timestamps are omitted until known. Correlate its Casework request id with Casework’s request, task, result, and result-feed resources. After durable submission, submission.recoveryDeadline reports the fixed recovery deadline. It bounds submission and cancellation retries, not the review itself: once the authority accepts a review, it may stay pending for as long as the authority holds it. Automatic application jobs report application.attempts from 0 through 1000; application.nextAttemptAt is present only while the job is queued or applying and is omitted once it is blocked or applied. An applied job includes application.receiptRecovered: true when the Base Registry Engine reconstructed the receipt by discovering the source request in its already-applied state. Omission means the Base Registry Engine completed through the apply response; an idempotent retry may have replayed that response, so omission does not prove that the source applied the request for the first time. Use application.state with recovery.code to choose the supported operator action, and do not query the database directly.

A 409 or 412 from application schedules another discovery after 5 seconds, then doubles the delay up to 5 minutes. A failed business precondition or row boundary therefore does not consume the retry budget in a tight loop. The worker keeps the same job and idempotency key, and each delayed discovery and application still checks current source authority.

recovery.codeOperator action
remote-uncertain, cancellation-uncertain, result-lookup-uncertain, source-precondition-changedWait through application.nextAttemptAt where present and re-read the request. The durable worker retries automatically. A later pending answer or a reconciled result clears result-lookup-uncertain.
token-unavailableRestore the configured review-authority credential provider, then let the durable worker retry.
submission-recovery-expiredReconcile the Casework request by the recorded idempotency and digest bindings. If the authority never received it, bregctl review-recovery resubmit submits the same retained request again; otherwise start a new proposal.
remote-refusedCorrect the producer, policy, or submitted contract reported by the review authority before submitting a new proposal.
cancellation-recovery-expired, cancellation-attempts-exhaustedReconcile the exact Casework request and cancellation idempotency key. If cancellation did not complete, correct the binding before starting a fresh proposal.
result-expiredStart a new review; the authority no longer promises the result payload.
result-unknown-to-authorityThe authority answered the result lookup with an empty 404: it no longer holds the accepted review, typically because the review environment was restored from an older backup or replaced. The engine keeps polling, after every review the authority still knows, until the attempt budget runs out. Once the authority is back in its intended state, run bregctl review-recovery resubmit to submit the same retained request again, or bregctl review-recovery close to stop waiting.
result-poll-attempts-exhaustedThe result lookup failed, or answered an empty 404, until the attempt budget ran out; a review the authority keeps answering as pending is never failed. Reconcile the accepted Casework request by its retained binding. If the authority lost the review, bregctl review-recovery resubmit submits the same retained request again. If the authority still holds a result, repair the result endpoint and start a new proposal; the retained binding identifies the review to close.
operator-closedAn operator closed the review with bregctl review-recovery close. Start a new proposal, or run bregctl review-recovery resubmit if the authority can take the same request again.
executor-unconfiguredRestore the named executor in runtime configuration, then use the advertised authorized manual apply action for the same approved proposal.
executor-deniedCorrect the executor credential or application grant, restart with that binding, then run bregctl review-recovery retry-application for the exact request and proposal version. The approval must still be current.
source-action-unavailableRestore the compiled source apply action and its access profile, then use the advertised authorized manual apply action for the same approved proposal.
source-response-invalidRepair the source deployment so its read and apply responses match the compiled contract, then use the advertised authorized manual apply action.
application-attempts-exhaustedInspect the source and executor logs using the request and application ids, then apply manually or create a new proposal.

To recover an executor-denied job after correcting its binding:

Terminal window
bregctl --format json review-recovery retry-application \
--runtime-config /srv/registry/runtime.yaml \
--request-entity correction-request \
--request-id 00000000-0000-4000-8000-000000000001 \
--proposal-version 1

The report returns state: queued. Re-read the request’s review.application until it reports applied or a new recovery code. Recovery preserves the exact proposal and job idempotency key. It refuses an expired approval, a withdrawn or replaced proposal, a running job, and a block for any other reason.

Resubmit or close a review the authority lost

Section titled “Resubmit or close a review the authority lost”

A review environment restored from an older backup, or replaced by a fresh one, no longer holds the reviews the engine submitted to it. Two commands recover one exact request proposal version. Both need the migration authority of the runtime file, take the registry lock, and append one audit record that names the request only by its keyed record reference.

Terminal window
bregctl --format json review-recovery resubmit \
--runtime-config /etc/registry/breg/runtime.yaml \
--request-entity correction-request \
--request-id 00000000-0000-4000-8000-000000000001 \
--proposal-version 1

resubmit accepts an accepted review coded result-unknown-to-authority, or a failed one coded result-poll-attempts-exhausted, submission-recovery-expired, or operator-closed. It releases the recorded authority binding, resets the attempt counters, restarts the recovery deadline with the configured window, and queues the retained request for submission under its original idempotency key. An authority that still holds the request answers with the same binding; one that lost it creates the review again. A result the old binding delivers later is recorded as unmatched.

Resubmitting relies on the authority still honouring that idempotency key. Before you resubmit a review coded result-poll-attempts-exhausted or operator-closed, confirm at the authority that it no longer holds the review: an authority whose idempotency retention has lapsed opens a second review while a reviewer may still be working the first.

close accepts an accepted review without a result. It marks the submission failed with operator-closed and keeps the binding for reconciliation, and a result the authority delivers for it afterwards is refused. It does not contact the authority, so cancel the review there as well if it still exists.

Both refuse a withdrawn proposal, a review whose result is already recorded, and any other state with review_recovery.submission.ineligible, whose message names the reason, state, and code. resubmit also refuses once request retention erased the review request (request-erased), or when the request no longer awaits review for that proposal version (proposal-not-submitted).

Each operation writes an audit request entry to the bregctl companion audit file before it changes the submission, and its response after the change commits. An audit destination that refuses the request entry stops the operation before it changes anything. One that refuses the response after the commit is reported as review_recovery.recovery.unaudited: the recovery took effect, so read the submission’s state before retrying.

Retain the project sources, module locks, generated files, test receipt, and runtime file for each activation. Store secrets separately.

Turn encryption on over existing plaintext

Section titled “Turn encryption on over existing plaintext”

A successor package can turn encrypted: true on a field the registry already holds in plaintext. The apply seals the stored values through the engine-executed backfill the package’s reviewed migration declares, and the history choice the project authored in that migration decides what happens to the plaintext the database retains. The two choices and their trade-offs are in Field encryption for restricted fields; this section is the operator path around the apply, which Change an active registry prepares and applies.

Run the preflight before you apply, while the runtime file still names the active package:

Terminal window
bregctl field-encryption preflight \
--runtime-config /etc/breg/runtime.yaml \
--package /srv/registry/build-2/package

The preflight is read-only and value-free. It binds the same predecessor package an apply binds, refuses when the active revision is anything else, and reports counts per covered field: the live rows still carrying plaintext, the retained journal revisions that carry the field member, and the retained change-request snapshots, cached idempotency responses, and event payloads that mention the field. When a covered field declares a unique blind index, the preflight refuses when two of those plaintext values would normalize onto one index entry, naming the authored record identifiers, capped at 64, and never a value. Correct one of the named records and run the preflight again; the apply refuses the same collision before it seals anything.

After the successor package is active, write an owner-only JSON request file (mode 0600) carrying the operator reference and the reason alone:

{
"operatorReference": "approved-maintenance-003",
"reason": "flip declared erase-and-rebaseline"
}

Then run the erase-history lifecycle:

Terminal window
bregctl field-encryption erase-history \
--runtime-config /etc/breg/runtime.yaml \
--request-file /srv/registry/private/erase-history.json

The request file names no record, entity, or field: the scope is the recorded erase-and-rebaseline flips themselves, so a document that names records or fields is refused rather than reinterpreted, and a request file that is not owner-only is refused before any connection is opened. For every record still holding pre-flip plaintext, the lifecycle erases its retained history through the same path history erase uses and restores snapshot coverage with one rebaseline, then writes one audit record that carries counts, not values. That rebaseline blocks reads and writes of every entity until it commits, so run the lifecycle in the maintenance window the rebaseline outage guidance describes.

bregctl field-encryption keygen --output <absolute-file> writes one fresh base64 data key for the local file provider with owner-only permissions, never prints it, and refuses to overwrite an existing file.

The container serves only a package that activation already recorded, and the runtime document you activated with on the operator host is not the one the container serves. Write a second runtime document for the container, changing exactly four things from the host’s: listener.bind, secretProviders.file.root, package.root, and the database hostname the resolved connection URL names, because the container mounts its own paths and reaches the database over its own network.

listener:
bind: 0.0.0.0:8080
secretProviders:
file:
root: /etc/breg/secrets
package:
root: /var/lib/breg/package

Nothing restricts listener.bind to a loopback or private address the way metricsListener.bind is restricted, so the container can bind 0.0.0.0 and accept traffic from outside its network namespace; a host process that only ever serves 127.0.0.1 has no reason to. Mount the three configuration inputs read-only, add a writable audit volume owned by the container’s non-root user, and publish the listener port. Set BREG_IMAGE to the release image digest from Obtain the runtime before running:

Terminal window
docker run --rm \
-p 8080:8080 \
-v /srv/registry/container/runtime.yaml:/etc/breg/runtime.yaml:ro \
-v /var/lib/breg/build-1/package:/var/lib/breg/package:ro \
-v /etc/breg/container-secrets:/etc/breg/secrets:ro \
-v /var/lib/breg/audit:/var/lib/breg/audit:rw \
"$BREG_IMAGE"

Run bregctl verify and bregctl doctor against this exact runtime document before you rely on the container’s GET /ready, either from a host that reaches the same database and the same mounted package, or from the image as the next section shows.

The image carries the bregctl built from the same source as its breg, so you can plan, apply, and read the activation ledger from the image digest you serve, without installing bregctl on an operator host. Override the entrypoint with /usr/local/bin/bregctl and pass the same arguments Activate and serve uses, with the container’s paths:

Terminal window
docker run --rm \
--entrypoint /usr/local/bin/bregctl \
-v /srv/registry/container/runtime.yaml:/etc/breg/runtime.yaml:ro \
-v /var/lib/breg/build-1/package:/var/lib/breg/package:ro \
-v /etc/breg/container-migration-secrets:/etc/breg/secrets:ro \
-v /var/lib/breg/audit:/var/lib/breg/audit:rw \
"$BREG_IMAGE" \
--format json plan \
--runtime-config /etc/breg/runtime.yaml \
--package /var/lib/breg/package

Run apply the same way, replacing the last three lines with --format json apply --runtime-config /etc/breg/runtime.yaml --package /var/lib/breg/package, then read the ledger with --format json status --runtime-config /etc/breg/runtime.yaml. Add --initial to apply only when plan reports activation: initial, for the first package in an uninitialized database; a later package is applied without it. plan, apply, and status connect with the migration credential that database.migrationUrlRef names, so mount a secret directory that holds it beside the files the runtime document references, and keep that credential out of the serving container’s secret directory. Make the image’s non-root user, UID 65532, the owner of every secret file, as for the serving container, because the resolver refuses a file owned by anyone else. apply appends its audit entries to a file of its own beside audit.path, so the audit volume is mounted writable here too. A --backup binding file is read inside the container: mount it read-only and name its container path. In Kubernetes, run the same command as a Job whose container sets command: ["/usr/local/bin/bregctl"] and puts the subcommand and its flags in args.

The listener that serves the registry API never serves metrics. To expose them, add a second listener that only the operator’s network can reach:

metricsListener:
bind: 127.0.0.1:9464

The address must be a loopback or private address (an IPv4 private range, or IPv6 unique-local) with a non-zero port, and it must differ from listener.bind; the runtime file is otherwise refused with runtime_config.invalid_metrics_listener. The listener answers GET /metrics in the Prometheus text format and nothing else, and it carries no authentication, so keep it behind the scraper’s network boundary rather than a public one.

SeriesTypeLabels
breg_http_requests_totalcounterroute, method, status
breg_http_request_duration_secondshistogramroute, method, status
breg_anonymous_refusals_totalcounterroute, method, reason
breg_pool_connectionsgaugestate: max_size, size, available, waiting

route is the matched route template, such as /v1/records/placements/{record_id}, or unmatched; it never carries a record identifier. status is success, client_error, or server_error rather than the exact code. The label sets are closed, so a scrape cannot grow without bound, and no label carries a principal, a token, or a record value. Pool states are read from the connection pool at scrape time.

breg_anonymous_refusals_total counts requests that presented no credential and were refused before admission: a profile the caller cannot hold, a route it cannot see, an absent scope or purpose, or a query the route cannot parse. reason is one of nine fixed values, read_request_invalid, read_concealed, read_refused, revision_request_invalid, revision_concealed, revision_refused, mutation_concealed, mutation_refused, and action_refused, derived from the refusal itself rather than from anything the request carried. These refusals are counted rather than journaled, because a caller with no principal names nobody the journal could hold accountable; refusals of an authenticated principal are journaled, and so is every admitted request. Alert on a sharp rise in this counter the way you would on a rise in client_error responses: it is the signal the journal no longer carries.

An admitted anonymous read writes the same pre-I/O audit envelope as any admitted request, and the runtime enforces no request-rate limit of its own, so put a registry that admits anonymous reads behind an upstream rate limit.

SymptomNext move
test refuses the runtime fileCompare it with the serving file: same identity, schema-test database references, and an empty package root.
A secret reference is refusedThe file under the secret root must be a regular file owned by the running user, with mode 0400 or 0600, one link, and no symbolic link, and its name must follow the naming rule in Create the secret files.
package refuses the receiptThe receipt binds sources, baseline, and fingerprint. Rerun test for the exact candidate.
doctor reports startup.instance_claim.mismatchThe database is a copy the registry’s instance claim does not name, as after a logical restore. Adopt it once the original is stopped for good, as back up and restore the database describes.
doctor reports startup.database.uninitializedNo package was ever activated in this database. Run plan, then apply --initial, as Activate and serve shows.
doctor reports startup.database.pre_ledgerA release before the activation ledger activated this database. Rebuild the active project unchanged with this release’s bregctl package, point package.root at the rebuilt package, then plan and apply it once with --package naming the same directory and without --initial; that apply adopts the database into the ledger.
doctor reports startup.database.identity_mismatchThe database records another database id than identity.databaseId. Point the database URLs at the database the file names, or correct identity.databaseId.
doctor reports startup.package.not_activeThe ledger names another package as active than the one at package.root. Run bregctl status to see the active digest, then set package.root to that package, or plan and apply the package you meant to serve.
doctor reports startup.runtime_role.can_writeIn split role mode the runtime role can write the activation ledger or the registry state, for example through a grant, an ownership, a membership, or a trigger no compiled migration creates. Run bregctl plan --package DIR: its refusal names the object and the statement that removes it, such as DROP TRIGGER.
doctor reports startup.runtime_role.grants_missingThe runtime role lacks a grant the active package gives it, as after REASSIGN OWNED. Run bregctl apply --package DIR with the active package to reissue the grants.
doctor reports startup.role_mode.changedThe ledger records a split role activation, but the runtime file names one role for both. Restore the separate runtime role, or plan and apply the active package under one role, which records a role_change activation.
plan or apply reports apply.successor.roles_differThe runtime file names other database roles than the active activation serves with, and only a role change retires the recorded runtime role. Run plan and apply with the active package under the new roles, which records a role_change activation, then apply the successor.
doctor reports startup.oidc.refusedCheck the issuer, discovery reachability or the static JWKS document, the algorithm, and the leeway bound.
doctor reports startup.package.refused with the package compiler derivation failed after an engine upgradeThe upgraded compiler derives a different schema from the active package, for example a new reference index. Build and apply a successor package with the upgraded bregctl before starting the upgraded breg, as Change an active registry describes. diff --runtime-config reads the active package, and test and package accept its directory through --baseline-package; verify refuses it until the successor is active.
doctor or startup warns postgres.connections.pool_over_halfLower database.pool.maxSize or raise max_connections until every replica, operator command, and monitoring session fits; see Set a PostgreSQL baseline.
doctor or startup warns postgres.autovacuum.off or postgres.track_counts.offTurn the setting back on in the server configuration and reload PostgreSQL, then run ANALYZE in the registry database so planner statistics catch up.
The runtime file is refused with runtime_config.invalid_metrics_listenermetricsListener.bind must be a loopback or private address with a non-zero port, distinct from listener.bind.
Every token is refusedCompare the token’s header and claims with the token table; a list-valued aud or a missing principal claim refuses the token.
An operator command cannot reach PostgreSQLCheck the migration URL secret, the role names, and SSL_CERT_FILE for a private authority.
A command refuses a path that existsSome component of the path is a symbolic link. Name the real directory; on macOS pass /private/tmp rather than /tmp.