Registry stack documentation: machine-readable Markdown. Index of all pages: https://docs.registrystack.org/v/0.38.0/llms.txt Full corpus: https://docs.registrystack.org/v/0.38.0/llms-full.txt # Deploy a registry > Install the Base Registry Engine runtime, provision PostgreSQL, write the runtime configuration, test, package, and activate the first package, and expose metrics for a registry that is ready to serve. You have a project that passes `bregctl check --production` ([Build a production candidate](../../tutorials/build-a-breg-production-candidate/) gets you there) and want it serving. At the end of this page a `breg` process serves a package against PostgreSQL, answers `GET /ready`, and exposes its metrics to your scraper. This page covers the deploy phase only: from an empty PostgreSQL server to the first activated package. The sibling pages take over once the registry serves: [Bind webhook receivers](../breg-webhooks/) for the destinations the project declares, [Change an active registry](../breg-changes/) for successor packages, [Retain, erase, and audit](../breg-retention/) for history, retained Evidence uses, and the audit journal, and [Move data in bulk](../breg-data/) for imports and exports. Two roles take part. The author prepares a package from the project. The operator, holding the migration credential, activates the package and runs the server. One person may hold both roles on a pilot; the commands stay the same. A package carries no signature: the migration credential is what authorizes an activation, and the database records each one in its activation ledger, which `bregctl status` reads. Every command reads `--help`. Paths given to `--runtime-config`, `--credentials`, `--output`, and `--package` must be absolute. Many operator paths are resolved through held directory descriptors rather than by name: data export output and checkpoint, data import state and checkpoint, test receipt output and the credentials file `bregctl test` reads, package inputs and outputs, history request files, the reviewed migrations tree, project migrate writes, and authoring sources, including each module's SQL assets and planner scripts, which are read through the module directory the listing opened. `bregctl` resolves each component of these paths against the directory descriptor that holds it, then opens, creates, renames, and publishes through those held descriptors, so a directory replaced by a symbolic link after the path is validated cannot redirect the command. `--runtime-config` and `--package` are resolved by pathname in `registry-breg` instead, as is every `bregctl dev` path, and the local file `apply --backup =` names, which `registry-breg` opens with `O_NOFOLLOW` on its final component only. Name the real directory rather than a symbolic link: on macOS, `/tmp`, `/var`, `/etc`, and `/home`, among others, are reached through symbolic links at the root, so pass `/private/tmp` and `/private/var` for the descriptor-resolved paths above. The descriptor-based refusal is fail-closed: the Linux and macOS builds resolve every descriptor-based path this way, and a build for a platform that offers no equivalent kernel-enforced resolution refuses each one instead of falling back to resolution by pathname; the diagnostic code depends on the command, since each surface reports the refusal under its own code rather than one shared across all of them. {/* Evidence: crates/registry-bregctl/src/data_lifecycle.rs, resolve_write_destination(); crates/registry-bregctl/src/test_lifecycle.rs, preflight_output() and read_credentials(); crates/registry-bregctl/src/package_lifecycle.rs; crates/registry-bregctl/src/history_erasure_lifecycle.rs and history_rebaseline_lifecycle.rs; crates/registry-bregctl/src/reviewed_migrations.rs; crates/registry-bregctl/src/lib.rs, write_project_registry(), write_migration_files_with_fault(), read_bounded_source_file(), read_module_directory_names(), load_module_asset_files(), and path_diagnostic(); crates/registry-bregctl/src/safe_path.rs, SafeDir and SafeEntry; crates/registry-breg/src/runtime_config.rs, load_runtime_config_with_env(); crates/registry-breg/src/package.rs, reject_relative_symlinks(); crates/registry-breg/src/migration.rs, open_bound_backup(); crates/registry-bregctl/src/dev/mod.rs, project(); crates/registry-bregctl/src/apply_lifecycle.rs, parse_backup_arguments(); products/breg/scripts/test-adopter-workflow.sh. */} ## Obtain the runtime Install the binaries on the host that will serve, and on the operator host that holds the migration credential: ```sh curl -fsSL https://github.com/registrystack/registry-stack/releases/latest/download/breg-install.sh | bash breg --version bregctl --version ``` The installer verifies every downloaded binary against the release `SHA256SUMS` before anything reaches the install directory, and installs `breg` and `bregctl` together or not at all. The installer does not verify release authenticity. Replace `| bash` with `| less` to read it before running it on a host you operate. `BREG_INSTALL_DIR` selects the install directory, and the default is `~/.local/bin`. The `latest` URL follows the newest release; to pin one, run that release's `breg--install.sh` from `https://github.com/registrystack/registry-stack/releases/download//`, which installs its own tag and refuses another. For a higher-assurance installation, follow the [release verification procedure](https://github.com/registrystack/registry-stack/blob/main/release/VERIFY.md) for the selected release, then rerun the installer with `BREG_ASSET_DIR` naming the verified directory. Binaries are published for `linux-amd64`, `linux-arm64`, and `macos-arm64`. Testing, packaging, activation, and serving must use the same release of both binaries, and the documentation archive matching that release describes them. To run a source checkout instead, build both binaries from its root with the [maintained build helper](https://github.com/registrystack/registry-stack/blob/main/scripts/cargo-runtime-library-path.sh), which also sets the dynamic-library search path macOS needs, and use that one build for every step. Record the checkout revision with the candidate, because a source build's version string alone does not identify its source: ```sh . ./scripts/cargo-runtime-library-path.sh registry_cargo_build "$PWD" --locked -p registry-breg \ --features registry-breg/runtime -p registry-bregctl --bins ``` Each release publishes the container image `ghcr.io/registrystack/breg:`. The release manifest published beside the binaries, `registry-stack--release-manifest.json`, records the promoted digest of the image; pin that digest as `BREG_IMAGE` in [Serve the container image](#serve-the-container-image) rather than the movable tag. The release image is built on distroless nonroot for `linux/amd64`; see [platform support](../../explanation/known-limitations/#platform-support) for the artifact matrix. Its entrypoint is `/usr/local/bin/breg`, its default arguments are `--runtime-config /etc/breg/runtime.yaml`, and it exposes port 8080. Beside the runtime, the image carries the matching `bregctl` at `/usr/local/bin/bregctl`; the entrypoint stays `breg`. The image carries no shell, no writable directory, and no healthcheck subcommand. Mount a writable volume for the file audit destination; mount the runtime configuration and its package and secret files read-only under `/etc/breg`. The orchestrator probes `GET /health` or `GET /healthz` for liveness and `GET /ready` for readiness. Activation runs either on a separate operator host with the matching `bregctl`, or from the same image with its entrypoint overridden, as [Activate from the image](#activate-from-the-image) shows. {/* Evidence: crates/registry-breg/install.sh; scripts/cargo-runtime-library-path.sh, registry_cargo_build(); .github/workflows/release-candidate.yml, the breg installer rendering; release/scripts/release_candidate.py, BREG_RELEASE_MINIMUM_VERSION, BREG_RUNTIME_IMAGE_NAMES, OPERATOR_TOOL_MINIMUM_VERSION, and IMAGE_OPERATOR_TOOLS; release/docker/Dockerfile.breg; release/scripts/build-release-image.sh; crates/registry-breg/src/api/mod.rs; release/VERIFY.md. */} ## Provision PostgreSQL Base Registry Engine needs PostgreSQL 17 or newer, with TLS between the server and the database. A project with a `crs84-point` field also needs PostGIS. Create two login roles: a migration role that owns the schema and applies packages, and a runtime role the serving process uses. Keep the migration credential off the serving host when no activation is in progress. Two separate roles are the split role mode, and production uses it. The serving process then cannot write the activation ledger or the registry state, and both `apply` and startup refuse a runtime role that could, naming the grant, ownership, membership, or trigger and the statement that removes it. Naming the same role for both is the single role mode that `bregctl dev` and the quickstart use: the ledger still catches a wrong package or database, but not someone who holds that one credential. Startup logs which mode it serves in, and `plan` and `status` report it as `roleMode`. {/* Evidence: crates/registry-breg/src/postgres/migration_ledger.rs, RoleMode, from_roles; crates/registry-breg/src/postgres/roles.rs, RuntimeWriteAuthority, find_runtime_write_authority; crates/registry-breg/src/startup.rs, RuntimeWriteAuthority; crates/registry-bregctl/src/lib.rs, PlanSuccessReport, StatusSuccessReport. */} ```sql CREATE ROLE registry_migration LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOBYPASSRLS PASSWORD ''; CREATE ROLE registry_runtime LOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOBYPASSRLS PASSWORD ''; ``` Create two databases with the same shape: the serving database, and a schema-test database for `test`. The five managed schemas in the schema-test database must be empty when `test` starts, and `test` fills them without cleaning up after itself, so drop and recreate that database before every run; a run against a database that still holds the previous run's objects refuses with `schema-test database is not clean`. In each database, as an administrator: ```sql CREATE EXTENSION IF NOT EXISTS btree_gist; REVOKE ALL ON DATABASE registry FROM PUBLIC; GRANT CONNECT ON DATABASE registry TO registry_migration, registry_runtime; CREATE SCHEMA registry_internal AUTHORIZATION registry_migration; CREATE SCHEMA registry_data AUTHORIZATION registry_migration; CREATE SCHEMA registry_source AUTHORIZATION registry_migration; CREATE SCHEMA registry_derived AUTHORIZATION registry_migration; CREATE SCHEMA registry_context AUTHORIZATION registry_migration; REVOKE ALL ON SCHEMA registry_internal, registry_data, registry_source, registry_derived, registry_context FROM PUBLIC; ``` Base Registry Engine creates every table, function, and policy inside those five schemas from the package. It never creates a schema, an extension, or a role, so a missing prerequisite surfaces at `test` or `apply` rather than at runtime. For a spatial project, add a bounding-box role named after the runtime role with the suffix `__spatial_bbox`, and install PostGIS in a dedicated schema that neither registry role owns: ```sql CREATE ROLE registry_runtime__spatial_bbox NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOINHERIT NOBYPASSRLS; GRANT registry_runtime__spatial_bbox TO registry_migration WITH INHERIT FALSE, SET TRUE, ADMIN FALSE; -- in each database CREATE SCHEMA registry_spatial_ext AUTHORIZATION postgres; CREATE EXTENSION IF NOT EXISTS postgis WITH SCHEMA registry_spatial_ext; REVOKE CREATE ON DATABASE registry FROM PUBLIC, registry_migration, registry_runtime; REVOKE ALL ON SCHEMA registry_spatial_ext FROM PUBLIC; GRANT USAGE ON SCHEMA registry_spatial_ext TO registry_migration, registry_runtime, registry_runtime__spatial_bbox; ``` The maintained runtime validates these ownership boundaries when it prepares spatial storage. {/* Evidence: crates/registry-breg/src/postgres/schema.rs, refuse_existing_managed_objects() and prepare_schema_test_database_with_connections(); crates/registry-breg/src/postgres/roles.rs. */} ### Set a PostgreSQL baseline PostgreSQL's defaults let it start on almost any machine; they are not sized for serving. On a dedicated database host, begin from these values and adjust them against your own measurements. `shared_buffers` and `max_connections` take effect only after a server restart. | Setting | Starting point | Why | | --- | --- | --- | | `shared_buffers` | 25% of memory on a host with 1 GB or more; a smaller share below that | PostgreSQL's own page cache. Past 40% it rarely helps, and a larger value usually wants a larger `max_wal_size`. | | `effective_cache_size` | `shared_buffers` plus the memory the operating system can spend caching PostgreSQL files | A planner estimate that allocates nothing; a higher value makes index scans more likely. | | `work_mem` | The 4 MB default, until measurements show sorts or hashes spilling to temporary files | It applies to each sort or hash operation in each session, so total use can reach many times the value. | | `maintenance_work_mem` | Above the 64 MB default when memory allows | `VACUUM` and `CREATE INDEX` use it, and autovacuum may take up to `autovacuum_max_workers` times the value. | | `random_page_cost` | Below the 4.0 default, toward `seq_page_cost` (1.0), when the database fits in server memory | Lowering it relative to `seq_page_cost` makes the planner prefer index scans. | **Budget connections.** Every `breg` replica opens up to `database.pool.maxSize` runtime connections. Operator commands such as `bregctl apply`, `bregctl doctor`, and `bregctl data import` open their own sessions, and so do monitoring agents and administrators. All of them must fit in the connections PostgreSQL leaves for ordinary roles: `max_connections` minus `superuser_reserved_connections` minus `reserved_connections`. At startup `breg` logs, and `bregctl doctor` reports, the numbers it compared under `postgres.connections.budget`. When one replica's pool alone may take more than half of the usable connections, the advisory is the `postgres.connections.pool_over_half` warning instead. The check sees one replica; add the others yourself. **Find sessions by name.** Runtime sessions report `breg` as their `application_name` and operator commands report `bregctl`. A name in the connection URL is kept, so `?application_name=breg-a` tells replicas apart in `pg_stat_activity`: ```sql SELECT application_name, state, count(*) FROM pg_stat_activity WHERE datname = 'registry' GROUP BY application_name, state; ``` Runtime sessions also carry `idle_in_transaction_session_timeout` of 120 seconds as a startup option placed after any options in the URL. PostgreSQL ends a runtime session that sits idle inside an open transaction for longer, the transaction rolls back, and the pool opens a replacement. Operator command and migration sessions carry no such bound, and no session carries `idle_session_timeout`. Runtime sessions also carry `jit=off` as a startup option placed before any options in the URL, because row-level security inflates the planner's cost estimate for short collection pages past the JIT thresholds and compiling such a plan costs more than running it. To turn JIT back on, add `-c jit=on` to the URL `options`; operator command and migration sessions keep the server default. **Pool in session mode only.** Base Registry Engine holds session-level advisory locks and sets session settings, so a pooler in front of it must hand each client one server connection until the client disconnects. Transaction pooling is not supported. PgBouncer raises an error for a startup parameter it does not track unless `ignore_startup_parameters` lists it, and ignores a listed one; with `options` in that list it ignores the unknown parameters inside `options`, including the idle bound and `jit=off` ([PgBouncer configuration](https://www.pgbouncer.org/config.html#ignore_startup_parameters)). When a pooler drops the startup options, set both on the runtime role, where they then also apply to operator commands that connect as that role: ```sql ALTER ROLE registry_runtime SET idle_in_transaction_session_timeout = '120s'; ALTER ROLE registry_runtime SET jit = off; ``` **Record statement timings.** Add `pg_stat_statements` to `shared_preload_libraries`, restart PostgreSQL, and then, as an administrator in the registry database: ```sql CREATE EXTENSION IF NOT EXISTS pg_stat_statements; ``` Base Registry Engine's catalog checks accept the extension either way, but the `postgres.pg_stat_statements.unavailable` advisory only stops once the preload above actually took effect. PostgreSQL lets `CREATE EXTENSION` succeed without `shared_preload_libraries`, then refuses every query against the view, so skipping the restart leaves `doctor` advising, with a message naming the module as installed but not loaded. It also keeps advising while `pg_stat_statements.track` is `none` or `compute_query_id` is `off`, because the module then records nothing. Only superusers and roles with the privileges of `pg_read_all_stats` see the statement text of other roles. **Keep maintenance running.** Leave `autovacuum` and `track_counts` on; with either off, `breg` and `doctor` warn, because dead rows accumulate and planner statistics go stale. After a large [`bregctl data import`](../breg-data/#import-with-a-checkpoint), run `ANALYZE` in the registry database as an administrator so the planner sees the new rows before autovacuum reaches them. Take a backup before every activation, as [Activate the successor](../breg-changes/#activate-the-successor) describes. On a managed service, the extension lists of Amazon RDS for PostgreSQL, Azure Database for PostgreSQL flexible server, and Google Cloud SQL for PostgreSQL each include `btree_gist`, `pg_stat_statements`, and PostGIS 3.5 or newer for PostgreSQL 17 (checked 2026-09-25). Set the parameters in the table, and `shared_preload_libraries`, through the provider's parameter settings; `doctor` reads the values the server actually runs with. {/* Evidence: crates/registry-breg/src/postgres/baseline.rs, advise() and connection_budget(), a_pool_over_half_of_usable_connections_warns_and_half_does_not() and vacuum_settings_that_are_off_warn(); crates/registry-breg/src/postgres/config.rs, name_session(), with_idle_in_transaction_session_timeout(), with_jit_disabled(), jit_is_disabled_before_operator_options_so_an_operator_can_enable_it(), sessions_carry_the_process_name_unless_the_url_names_one(); crates/registry-breg/src/startup.rs, SERVER_IDLE_IN_TRANSACTION_TIMEOUT and prepare_database_startup(); crates/registry-breg/src/mutation/action.rs, acquire_hook_proposal_lock(); crates/registry-breg/src/postgres/interlock.rs; crates/registry-breg/tests/postgres_kernel.rs, runtime_sessions_are_named_and_bounded_across_recycling and idle_in_transaction_bound_rolls_back_and_the_pool_recovers; crates/registry-breg/tests/postgres_startup.rs, prepared_server_sessions_are_named_bounded_and_pg_stat_statements_stays_unavailable_until_preloaded. */} ## Write the runtime configuration The runtime file binds one package to one database, one token issuer, and one listener, plus an optional private metrics listener. It is a deployment artifact: keep it with the operator, outside the authoring project, and never put a credential in it. Values such as `secret:file/` resolve to owner-only files under the file provider root; `secret:env/` reads an environment variable when `secretProviders.environment` is declared. ```yaml apiVersion: registry.registrystack.org/breg-runtime/v1alpha1 kind: BRegRuntimeConfig listener: bind: 127.0.0.1:8080 publicOrigin: https://registry.example.org identity: environment: production instanceId: civil-registry-1 databaseId: civil-registry-db-1 databaseInitializationEnvironment: production secretProviders: file: root: /etc/breg/secrets database: runtimeUrlRef: secret:file/runtime-database-url migrationUrlRef: secret:file/migration-database-url pool: maxSize: 8 roles: migration: registry_migration runtime: registry_runtime package: root: /var/lib/breg/packages/build-1/package expectedDigest: sha256: authentication: oidc: issuer: https://issuer.example.org audience: breg allowedAlgorithm: ES256 accessTokenType: at+jwt scopeClaim: scope scopeSeparator: " " allowedClients: [registry-console] deniedKids: [] maxTokenLifetimeSeconds: 300 leewayMilliseconds: 30000 jwksSource: kind: discovery authorityClaims: principal: registry_principal purpose: registry_purpose audit: hashKeyRef: secret:file/audit-key destination: file path: /var/lib/breg/audit/breg.jsonl cursor: secretRef: secret:file/cursor-key eventDestinations: {} ``` `package.expectedDigest` is optional. When you set it, copy the `packageDigest` reported by the successful `bregctl package` publication. Startup checks that digest before it checks that the database is the one `identity.databaseId` names and has activated this package. Rebuild a candidate with `bregctl package`; do not edit `SHA256SUMS`, `REVISION`, or files below `package.root` in place. {/* Evidence: crates/registry-breg/src/runtime_config.rs, RuntimeConfig::verify_package_envelope; crates/registry-breg/src/package.rs, PreparedPackage::publish_to_directory_with_revision; crates/registry-breg/src/startup.rs, prepare(). */} `publicOrigin` may include a deployment path prefix, such as `https://registry.example.org/registry-a`. Discovery, paging, and schema links preserve the configured prefix and never derive their authority from request headers. | Section | What it binds | | --- | --- | | `listener` | The bind address and the optional public origin that generated documents cite. The runtime reads neither peer addresses nor forwarded headers, so client addresses and TLS termination are enforced at your proxy. | | `identity` | The environment, instance, and database this file may serve. `environment` must equal `databaseInitializationEnvironment`. The first `apply` records `databaseId` in the database; every later `plan`, `apply`, and startup refuses a file that names another. `instanceId` names the `source` of every webhook event this runtime captures, and the delivery worker dead-letters a stored event whose source names another instance, so startup refuses a changed `instanceId` while pending deliveries were captured under the previous one; keep the previous `instanceId` until they drain, then change it. | | `secretProviders` | The file root (owner-only files) and, when declared, the environment provider. | | `database` | Secret references for the runtime and migration connection URLs, pool bounds, and the two role names the package's policies are written for. | | `package` | The package directory the activation ledger names as active and, optionally, the `expectedDigest` it must carry. | | `authentication` | The OpenID Connect verifier and the names of the claims that carry the caller's principal and purpose. | | `audit` and `cursor` | The audit destination and keyed-reference secret, and the secret that signs pagination cursors. Changing the audit key changes future pseudonyms; changing the cursor secret invalidates existing cursors. | | `eventDestinations` and `eventDelivery` | One binding per webhook destination the package declares, and payload retention; see [Bind webhook receivers](../breg-webhooks/). | | `evidenceProviders` | One binding per Evidence provider the package declares, keyed by provider ID: `baseUrl`, `trustBindingId`, `trustedJwksRef`, `revokedKeyIds`, optional `caBundleRef`, and exactly one of `tokenRef` or a refreshing `privateKeyJwt` credential. The script cannot replace these bindings; see [Governed registry actions](../../explanation/governed-registry-actions/). | | `fieldEncryption` | Key custody for encrypted fields; declare it when the project encrypts a field. The optional `provider` binding's `kind` member selects the custodian. `kind: transit` requires `unixSocketPath`, `mount`, and `keyName`, with an optional `timeoutMilliseconds` (default 5000, at most 30000). `kind: localFile` requires `dekRef`, a `secret:file/` reference to one base64 data-key file, and is for local assurance only. The model and its limits are in [Field encryption for restricted fields](../../explanation/breg-field-encryption/). | | `attachmentStorage` | PostgreSQL by default, or an operator-bound S3-compatible backend for request attachments; see [Attachment storage](../../reference/breg-configuration/#attachment-storage). | | `attachmentVerification` | Optional asynchronous HTTP verifier; content stays quarantined until approved. Configure it before the first attachment upload; see [External attachment verification](../../reference/breg-configuration/#external-attachment-verification). | | `operationalTimeouts` | Optional HTTP request, record lock, migration lock, migration statement, and shutdown grace bounds. | | `metricsListener` | Optional second binding, private to the operator, that serves `GET /metrics`; see [Scrape metrics](#scrape-metrics). | The `fieldEncryption` block is optional; its `provider` binding names the custodian: ```yaml fieldEncryption: provider: kind: transit unixSocketPath: /var/run/vault/transit.sock mount: transit keyName: breg-field-dek timeoutMilliseconds: 5000 ``` An encrypted field moves part of the registry's availability into key custody, so plan the Transit binding's custody before the first encrypted field is activated: :::caution[Losing the Transit key loses the sealed values] Fields the project declares `encrypted` are sealed under a data-encryption key that Transit wraps and only Transit unwraps. If startup or restart cannot reach Transit or unwrap the key, Base Registry Engine stops before serving any entity. After a successful start, an envelope that cannot be opened fails the affected encrypted-field read or write with the HTTP `503` problem `runtime.field_encryption.unavailable`; sibling entities without encrypted fields keep serving. Base Registry Engine stores only the Transit-wrapped key or a nonsecret identifier for a local key, and offers no recovery, so escrow the Transit key before enabling. The local file provider is development custody: startup refuses it unless the runtime file's `identity.databaseInitializationEnvironment` is `local`, and `bregctl doctor --runtime-config ` reports the same refusal. ::: {/* Evidence: crates/registry-breg/src/field_encryption.rs, RawFieldEncryptionConfig, RawFieldEncryptionProvider, FieldEncryptionConfig::from_raw(), and FieldEncryptionService::activate() and FieldEncryptionService::open_existing(); crates/registry-breg/src/problem.rs, RuntimeFieldEncryptionUnavailable; crates/registry-breg/src/api/mod.rs, field_encryption_refusal(); crates/registry-breg/src/startup.rs, FieldEncryptionCustody; crates/registry-bregctl/src/doctor.rs, startup_diagnostic(). */} `issuer` must be an `https` URL without credentials, query, or fragment; plain `http` is accepted only on an IPv4 loopback host, for local development or an issuer or proxy the operator runs on the same host. Use an `https` issuer in production. `audience` is at most 512 characters. `jwksSource` takes one of three kinds: `discovery` (the default) reads the key set location from the issuer's discovery document, `uri` names it directly with a `uri` member under the same URL rule, and `static` pins a document through `documentRef`. `leewayMilliseconds` must be a whole number of seconds and at most 300000. Static keys load at startup; rotate them through a configuration change and restart: ```yaml jwksSource: kind: static documentRef: secret:file/ ``` Tokens signed with a key absent from the pinned document are refused with the value-free `authentication.refused`. Search the server's JSON logs for `unknown or disallowed key identifier` in `fields.message`, with logging set to `warn` or `info`. The server emits this warning at most once per minute per authenticator. This warning applies to both static and discovery sources. An arbitrary token or a deliberately denied key can cause the same warning, so the warning alone does not establish provider key rotation. If fresh logins fail and you independently confirm that the provider rotated its keys or regenerated them after state loss, recover a static pin with these steps: :::caution[Confirm the trust change] The replacement document becomes the anchor for bearer verification. Confirm the provider's current keys through its trusted configuration and operator before replacing the pin. ::: 1. Fetch the provider's current JWKS document from its published `jwks_uri` over TLS, the same way the initial pinning did. Confirm the keys with the provider operator if the rotation was not announced. 2. Prepare and review a compatible public-key subset of that document. Keep only signing keys for your configured `allowedAlgorithm`, with unique `kid` values absent from `deniedKids`. Preserve their public key material and identifiers. Do not relabel algorithms or turn encryption keys into signing keys. The document must contain only a `keys` array with 1 to 128 keys and fit within both the 65536-byte secret limit and `jwksCache.maxDocumentBytes` (65536 bytes by default). Each key must use the matching public-key shape and only the members `kty`, `kid`, `alg`, `use`, `key_ops`, `crv`, `x`, `y`, `n`, and `e` that apply to that shape. Remove certificate metadata such as `x5c`; never include private key material. If present, `use` must be `sig` and `key_ops` must be exactly `["verify"]`. Review the resulting subset with the provider operator before use. 3. Replace the secret that `documentRef` names. For `secret:file/`, replace the file while keeping the [secret-file rules](#create-the-secret-files). For `secret:env/`, when `secretProviders.environment` is enabled, replace the injected environment value in your service configuration with the document bytes. Keep the value out of command arguments and logs. 4. Restart `breg`; the static document is read only at startup. If startup refuses the document, recheck its members, key shapes, algorithm, denied keys, and size before retrying. 5. Verify with a fresh provider login that authentication succeeds again, and that tokens signed by the removed keys are still refused. If fresh logins are still refused, confirm that every serving instance restarted with the intended `documentRef`, then check the issuer, audience, algorithm, token type, and key policy with the provider operator. Keep verification enabled. Restore a previous pin only if its keys remain trusted and the provider confirms they are still valid; never restore a revoked key to end an outage. If a token signed by a removed key is accepted, stop routing protected traffic to the affected instance. Check for an old process or another serving instance using the previous configuration, and escalate the trust discrepancy to the deployment and identity-provider operators. Resume traffic only after the intended pin accepts current keys and refuses removed keys. With `kind: discovery`, the runtime reads the issuer's discovery document and caches its JWKS under the optional `jwksCache` bounds. {/* Evidence: crates/registry-breg/src/runtime_config.rs, RuntimeConfig and validate_static_jwks(); crates/registry-breg/src/main.rs, initialize_logging_filter(); crates/registry-breg/tests/runtime_config.rs; crates/registry-breg/src/auth.rs, RegistryAuthenticator::authenticate; crates/registry-breg/tests/http_auth.rs, static_jwks_key_refusal_is_bounded_and_repinning_restores_authentication; crates/registry-platform-config/src/secrets.rs, SecretReference; products/breg/generated/runtime/runtime.schema.json. */} ### Create the secret files Create the files the example references as the user that runs `breg` and the operator commands, because the resolver refuses a file owned by anyone else. Each must be a regular file with mode `0400` or `0600` and no other link or symbolic link to it, and its name starts with a lowercase letter and uses lowercase letters, digits, `.`, `_`, and `-`. The bytes are used exactly as written, neither trimmed nor decoded, so write them without a trailing newline: ```sh install -d -m 0700 /etc/breg/secrets (umask 077; openssl rand -hex 32 | tr -d '\n' > /etc/breg/secrets/audit-key) (umask 077; openssl rand -hex 32 | tr -d '\n' > /etc/breg/secrets/cursor-key) (umask 077; printf '%s' 'postgresql://registry_runtime:@db.example.org:5432/registry' \ > /etc/breg/secrets/runtime-database-url) (umask 077; printf '%s' 'postgresql://registry_migration:@db.example.org:5432/registry' \ > /etc/breg/secrets/migration-database-url) ``` The audit key and the cursor secret each need at least 32 bytes; 64 hexadecimal characters satisfy both. Percent-encode a password that carries reserved characters. The user in each URL must equal the role named under `database.roles`, and the runtime requires TLS on both connections itself, so the URL needs no `sslmode` parameter and a server without TLS refuses the connection. {/* Evidence: crates/registry-platform-config/src/secrets.rs, read_secret_file(), validate_file_metadata(), and valid_file_name(); crates/registry-platform-audit/src/lib.rs, MIN_AUDIT_SECRET_BYTES; crates/registry-breg/src/cursor.rs, ROOT_SECRET_MIN_BYTES; crates/registry-breg/src/runtime_config.rs, database_connection_config_for(); crates/registry-breg/src/postgres/config.rs, require_tls_config(). */} ## What a token must carry Base Registry Engine verifies a bearer access token before it reads any claim. Configure the issuer so that every token for this deployment satisfies the table. Fixture claims from `journeys.yaml` are never accepted by a running server. | Token part | Requirement | | --- | --- | | `alg`, `typ`, `kid` headers | `alg` equals `allowedAlgorithm`; `typ` matches `accessTokenType` case-insensitively, and when `accessTokenType` names the RFC 9068 access-token media type (`at+jwt` or `application/at+jwt`), either spelling of that one type is accepted; `kid` is present in the JWKS and absent from `deniedKids`. | | `iss` | Equals `issuer`. | | `aud` | A string equal to `audience`, or an array of 1 to 16 distinct nonempty strings containing that exact resource audience. | | `exp`, `iat`, `nbf` | The lifetime is at most `maxTokenLifetimeSeconds`; clock skew up to `leewayMilliseconds` is tolerated. | | `azp` or `client_id` | Listed in `allowedClients` when that list is not empty. | | `scope` | The single claim named by `scopeClaim` covers every required permission. It is a string split on `scopeSeparator`, or a JSON array of permission strings. Other shapes are refused; permissions from several claims are not merged. | | principal claim | The claim named by `authorityClaims.principal` carries the identity recorded in audit and workflow decisions. It must match each authenticated profile's `principalClaim`. | | purpose claim | The claim named by `authorityClaims.purpose` is required when the profile lists `requiredPurposes`. | | assignment claims | Each row boundary or claim-backed lookup names its own claim. An `equals` boundary needs a scalar; an `in` boundary needs a JSON array. An identity claim does not imply district or team assignments. | Set `maxTokenLifetimeSeconds` to your deployment's chosen limit, up to 7200 seconds. The limit bounds the accepted difference between issuance and expiry; increasing it extends how long an already issued token may carry old permissions. Match the issuer's token settings and test both accepted and refused lifetimes. Use a resource audience dedicated to this registry API, distinct from your application's login client identifier. Obtain access tokens for that resource; an ID token issued to the login client is not an API credential. ### Choose identities that survive operations Select the principal claim explicitly. `sub` is supported when selected in both the runtime and profiles; the server never falls back to another identity claim if the selected claim is missing. An issuer's `sub` usually belongs to that issuer's identity namespace. For an issuer migration, prefer an institutional identifier carried in a custom claim by both issuers, and document who guarantees that its values remain unique, stable, and never reassigned. Base Registry Engine treats the selected value as an opaque identity and does not link accounts across issuers. Changing it can change whether someone is recognized as a submitter or a previous reviewer. Use the same principal claim in an ownership row boundary when the record's owner field stores that identifier. Use separate assignment claims for district, tenant, or team membership. Configure the issuer to derive these claims from trusted assignments rather than caller-supplied values. Configure a compatible OAuth issuer for registered machine clients. A client-credentials flow represents a service, not an individual signing in. Human login, account lifecycle, and multifactor authentication belong to your identity provider and application. When replacing an issuer, verify the same permitted and refused requests with real tokens before changing the serving configuration. Compare issuer, resource audience, token headers, permission source, principal identity, purpose, and assignment claim shapes. {/* Evidence: crates/registry-breg/src/auth.rs; crates/registry-breg/tests/http_auth.rs; crates/registry-breg/src/runtime_config.rs; crates/registry-platform-oidc/src/lib.rs. */} ### Rotate signing keys and handle issuer outages With discovery, `jwksCache.cacheTtlSeconds` defaults to 600 seconds. An unfamiliar `kid` can trigger a refresh, subject to `refreshCooldownSeconds` (30 seconds by default) and the bounded negative cache. Publish a new public key before issuing tokens with it, then verify a token using that key against the registry. Keep the previous public key available while its legitimate tokens remain valid. Removing a key from the issuer does not immediately remove an already cached key from the registry. During a fetch failure, a known cached key may remain usable until its age reaches the cache TTL plus `outageToleranceSeconds`, which defaults to 900 seconds. The default total allowance is therefore 1500 seconds from the last successful key-set fetch, not from the start of the outage. Unknown keys cannot use this allowance, and the server cannot obtain keys for a cold start from an unavailable issuer. After the allowance, verification requiring those stale keys fails until fetching succeeds. Token expiry and every other verification rule continue to apply during the allowance. Short token lifetimes bound how long issued permission and assignment claims remain usable. Disabling an account or changing a group at the issuer does not update claims in an already issued token. For a compromised signing key, add its identifier to `deniedKids` and restart every serving instance with that configuration. That rejects all tokens signed with the denied key, including otherwise legitimate tokens. Static JWKS documents also require a configuration change and restart to rotate. [Rotate credentials, keys, certificates, and trust](../advanced/rotate-credentials-and-trust/) covers related rotation boundaries for Relay and Evidence Gateway. ### Change the issuer without losing request context The runtime trusts one configured issuer and reads its configuration at startup. Plan a coordinated restart and client token change; one process does not provide an overlap period accepting both issuers. Keep the selected principal values stable if existing ownership and review decisions should continue to recognize the same actors. Test new tokens before the serving cutover, including refused permissions and assignment shapes. Resolve uncertain writes before changing issuer mappings, profiles, or packages. After a timeout, retry the same request with its original idempotency key and the same effective principal, profile, purpose, and row context. A replacement token can satisfy those conditions; changing identity or authority context can produce `idempotency.conflict` instead of a replay. The binding also covers the request and the activation that serves it, so issuing a fresh key blindly can duplicate a write whose first response was lost. [Mutation retry rules](../../reference/breg-api/#idempotency) explain the request contract. Pagination cursors bind the principal, profile, purpose, row scope, query, projection, and compiled registry identity. They also expire, with `cursor.maxAgeSeconds` defaulting to 300 seconds. A token renewal alone does not require starting over when the effective context is unchanged, but changed mappings, policy, package, or cursor secret can invalidate an existing cursor. Restart the query from its first page when that happens; do not treat a cursor as portable authority between deployments. {/* Evidence: crates/registry-platform-oidc/src/lib.rs, JwksFetcher and tolerated_age(); crates/registry-breg/src/runtime_config.rs, JwksCacheConfig and DEFAULT_CURSOR_MAX_AGE_SECONDS; crates/registry-breg/src/startup.rs, prepare(); crates/registry-breg/src/auth.rs; crates/registry-breg/src/idempotency.rs, canonical_claim_context(); crates/registry-breg/src/cursor.rs, CursorBinding; crates/registry-breg/src/api/mod.rs, cursor_binding(). */} ## Test the candidate `test` compiles the project, applies it to the empty schema-test database, measures the resulting schema, runs every journey in `tests/journeys.yaml` over real HTTP with real tokens, and writes a receipt that `package` later binds to. Prepare three inputs: 1. A test runtime file: the same document as the serving one, with `database` pointing at the schema-test database and `package.root` an empty directory. 2. Real access tokens from your issuer for each journey principal, stored as owner-only files under the secret root. 3. A credentials document binding every journey step to a token or to anonymous access: ```yaml apiVersion: registry.registrystack.org/breg-schema-test-credentials/v1 kind: SchemaTestCredentials bindings: - journeyId: record-lifecycle stepId: create-record credential: type: bearer tokenRef: secret:file/schema-test-token - journeyId: record-lifecycle stepId: get-record credential: type: bearer tokenRef: secret:file/schema-test-token - journeyId: record-lifecycle stepId: list-records credential: type: anonymous ``` Then run the test: ```sh bregctl --format json test ./my-registry \ --runtime-config /srv/registry/runtime-test.yaml \ --credentials /srv/registry/schema-test-credentials.yaml \ --output /srv/registry/schema-test-receipt.json ``` The report names the candidate `registryRevision`, the measured `schemaFingerprint`, and the journeys that passed. Keep the receipt and the fingerprint with the candidate: `package` refuses a receipt taken for different sources or a different baseline. A journey failure reports the journey and step without record values. {/* Evidence: crates/registry-bregctl/src/test_lifecycle.rs; crates/registry-bregctl/src/lib.rs, TestArgs; crates/registry-breg/src/fixtures.rs. */} ## Package `package` seals and publishes the deployable package directory in one step: ```sh bregctl --format json package ./my-registry \ --test-receipt /srv/registry/schema-test-receipt.json \ --output /srv/registry/build-1 ``` The schema fingerprint comes from the test receipt; `--schema-fingerprint` states it explicitly when you want the command to fail on any other value. The report's `packageDigest` names the package, and the package directory is `build-1/package`. The command refuses an output directory that already holds a published package, so each candidate needs a build directory of its own. {/* Evidence: crates/registry-bregctl/src/package_lifecycle.rs; crates/registry-bregctl/src/lib.rs, PackageArgs; crates/registry-breg/src/package.rs; products/breg/scripts/test-adopter-workflow.sh. */} ## Activate and serve Write the serving runtime file with `package.root` pointing at `build-1/package` and, if you pin it, `package.expectedDigest` equal to the reported `packageDigest`. Rehearse the activation with the migration credential, then activate: ```sh bregctl --format json plan \ --runtime-config /etc/breg/runtime.yaml \ --package /srv/registry/build-1/package bregctl --format json apply \ --runtime-config /etc/breg/runtime.yaml \ --package /srv/registry/build-1/package --initial ``` `plan` makes the checks `apply` makes before it changes anything, under the same lock, and rolls all of them back: it writes nothing, appends no audit entry, and exits 0 with `pending: true` and `activation: initial` when `apply` would succeed, or 1 with the refusal `apply` would give. `--initial` activates the first package in an uninitialized database, and `apply` reports the `activationId` of the row it recorded in the activation ledger. `bregctl status` reads that ledger back without taking the lock: ```sh bregctl --format json status --runtime-config /etc/breg/runtime.yaml ``` It reports the `databaseId`, the `activePackageDigest` with its `activationId` and `registryRevision`, the `roleMode`, the schema fingerprint and maintenance state, and one `ledger` entry per activation, each with its outcome. Confirm the configuration before starting the process: ```sh bregctl verify --runtime-config /etc/breg/runtime.yaml bregctl doctor --runtime-config /etc/breg/runtime.yaml breg --runtime-config /etc/breg/runtime.yaml ``` `verify` opens no runtime dependency. It verifies the configured package's files against its sum file and, when set, `package.expectedDigest`, re-derives every generated artifact from the package's own source, and reports the package digest, the registry id, version, and revision plus inventory counts. It does not say whether the database is on that package; `status` and readiness do, and startup refuses to serve a package the ledger does not name as active. `doctor` opens every startup dependency without binding the listener: the runtime file, the package, the database connection and its readiness, the audit key and destination writability, the cursor key, the OIDC verifier's key material, the event destination bindings, the retained review authority and executor bindings, the authentication profile's claim mapping, accepted algorithms, and audience against the package this runtime serves, and the field-encryption provider and its stored key. It names the first dependency that refuses and stops there. A run that reaches the end lists every dependency it checked, then the PostgreSQL advisories from [Set a PostgreSQL baseline](#set-a-postgresql-baseline). Advisories never change the exit status. This run is against a local `bregctl dev` session, whose runtime file sets `database.pool.maxSize: 4` and whose PostgreSQL keeps its default connection settings and has no `pg_stat_statements` extension, so it also reports the statement-timings advisory: ```text 10 dependency checks passed. runtimeConfig pass package pass database pass audit pass cursor pass authentication.oidc pass eventDestinations pass reviewBindings pass authentication pass fieldEncryption pass PostgreSQL advisories: information postgres.connections.budget one replica's runtime pool takes at most half of the connections PostgreSQL leaves for ordinary roles poolMaxSize 4 maxConnections 100 superuserReservedConnections 3 reservedConnections 0 usableConnections 97 information postgres.pg_stat_statements.unavailable pg_stat_statements is not installed in this database, so per-statement timings are unavailable when diagnosing load ``` ```json { "ok": true, "command": "doctor", "checked": [ "runtimeConfig", "package", "database", "audit", "cursor", "authentication.oidc", "eventDestinations", "reviewBindings", "authentication", "fieldEncryption" ], "advisories": [ { "code": "postgres.connections.budget", "severity": "information", "message": "one replica's runtime pool takes at most half of the connections PostgreSQL leaves for ordinary roles", "observed": { "poolMaxSize": 4, "maxConnections": 100, "superuserReservedConnections": 3, "reservedConnections": 0, "usableConnections": 97 } }, { "code": "postgres.pg_stat_statements.unavailable", "severity": "information", "message": "pg_stat_statements is not installed in this database, so per-statement timings are unavailable when diagnosing load", "observed": {} } ] } ``` {/* Evidence: crates/registry-bregctl/src/lib.rs, PlanArgs, PlanSuccessReport, StatusArgs, StatusSuccessReport, ApplySuccessReport; crates/registry-breg/src/migration.rs, AlreadyActive; crates/registry-breg/src/startup.rs, ActivePackageMismatch. */} Start the process and check `GET /ready` before admitting traffic; `GET /health` reports liveness only. Restarting this same version with the same runtime file, active package, and database resumes the durable review submission, result reconciliation, and automatic application jobs. A restart does not recreate a Casework review and does not bypass current Base Registry Engine application authority. `doctor` and readiness report setup and dependency health. They are not a change-request workflow console. To trace one review, read the request under a profile granted `readableRequestFields: [review_state]` and inspect `data.request.review`. Its closed sections are `submission`, `result`, `delivery`, `application`, and `recovery`; identifiers and timestamps are omitted until known. Correlate its Casework request id with Casework's request, task, result, and result-feed resources. After durable submission, `submission.recoveryDeadline` reports the fixed recovery deadline. It bounds submission and cancellation retries, not the review itself: once the authority accepts a review, it may stay pending for as long as the authority holds it. Automatic application jobs report `application.attempts` from 0 through 1000; `application.nextAttemptAt` is present only while the job is queued or applying and is omitted once it is blocked or applied. An applied job includes `application.receiptRecovered: true` when the Base Registry Engine reconstructed the receipt by discovering the source request in its already-applied state. Omission means the Base Registry Engine completed through the apply response; an idempotent retry may have replayed that response, so omission does not prove that the source applied the request for the first time. Use `application.state` with `recovery.code` to choose the supported operator action, and do not query the database directly. A `409` or `412` from application schedules another discovery after 5 seconds, then doubles the delay up to 5 minutes. A failed business precondition or row boundary therefore does not consume the retry budget in a tight loop. The worker keeps the same job and idempotency key, and each delayed discovery and application still checks current source authority. {/* Evidence: crates/registry-breg/src/review_store.rs; crates/registry-breg/tests/postgres_review_executor.rs */} | `recovery.code` | Operator action | | --- | --- | | `remote-uncertain`, `cancellation-uncertain`, `result-lookup-uncertain`, `source-precondition-changed` | Wait through `application.nextAttemptAt` where present and re-read the request. The durable worker retries automatically. A later pending answer or a reconciled result clears `result-lookup-uncertain`. | | `token-unavailable` | Restore the configured review-authority credential provider, then let the durable worker retry. | | `submission-recovery-expired` | Reconcile the Casework request by the recorded idempotency and digest bindings. If the authority never received it, `bregctl review-recovery resubmit` submits the same retained request again; otherwise start a new proposal. | | `remote-refused` | Correct the producer, policy, or submitted contract reported by the review authority before submitting a new proposal. | | `cancellation-recovery-expired`, `cancellation-attempts-exhausted` | Reconcile the exact Casework request and cancellation idempotency key. If cancellation did not complete, correct the binding before starting a fresh proposal. | | `result-expired` | Start a new review; the authority no longer promises the result payload. | | `result-unknown-to-authority` | The authority answered the result lookup with an empty `404`: it no longer holds the accepted review, typically because the review environment was restored from an older backup or replaced. The engine keeps polling, after every review the authority still knows, until the attempt budget runs out. Once the authority is back in its intended state, run `bregctl review-recovery resubmit` to submit the same retained request again, or `bregctl review-recovery close` to stop waiting. | | `result-poll-attempts-exhausted` | The result lookup failed, or answered an empty `404`, until the attempt budget ran out; a review the authority keeps answering as pending is never failed. Reconcile the accepted Casework request by its retained binding. If the authority lost the review, `bregctl review-recovery resubmit` submits the same retained request again. If the authority still holds a result, repair the result endpoint and start a new proposal; the retained binding identifies the review to close. | | `operator-closed` | An operator closed the review with `bregctl review-recovery close`. Start a new proposal, or run `bregctl review-recovery resubmit` if the authority can take the same request again. | | `executor-unconfigured` | Restore the named executor in runtime configuration, then use the advertised authorized manual apply action for the same approved proposal. | | `executor-denied` | Correct the executor credential or application grant, restart with that binding, then run `bregctl review-recovery retry-application` for the exact request and proposal version. The approval must still be current. | | `source-action-unavailable` | Restore the compiled source apply action and its access profile, then use the advertised authorized manual apply action for the same approved proposal. | | `source-response-invalid` | Repair the source deployment so its read and apply responses match the compiled contract, then use the advertised authorized manual apply action. | | `application-attempts-exhausted` | Inspect the source and executor logs using the request and application ids, then apply manually or create a new proposal. | To recover an executor-denied job after correcting its binding: ```sh bregctl --format json review-recovery retry-application \ --runtime-config /srv/registry/runtime.yaml \ --request-entity correction-request \ --request-id 00000000-0000-4000-8000-000000000001 \ --proposal-version 1 ``` The report returns `state: queued`. Re-read the request's `review.application` until it reports `applied` or a new recovery code. Recovery preserves the exact proposal and job idempotency key. It refuses an expired approval, a withdrawn or replaced proposal, a running job, and a block for any other reason. {/* Evidence: crates/registry-breg/src/review_recovery.rs; crates/registry-breg/tests/postgres_change_requests.rs */} ### Resubmit or close a review the authority lost A review environment restored from an older backup, or replaced by a fresh one, no longer holds the reviews the engine submitted to it. Two commands recover one exact request proposal version. Both need the migration authority of the runtime file, take the registry lock, and append one audit record that names the request only by its keyed record reference. ```bash bregctl --format json review-recovery resubmit \ --runtime-config /etc/registry/breg/runtime.yaml \ --request-entity correction-request \ --request-id 00000000-0000-4000-8000-000000000001 \ --proposal-version 1 ``` `resubmit` accepts an accepted review coded `result-unknown-to-authority`, or a failed one coded `result-poll-attempts-exhausted`, `submission-recovery-expired`, or `operator-closed`. It releases the recorded authority binding, resets the attempt counters, restarts the recovery deadline with the configured window, and queues the retained request for submission under its original idempotency key. An authority that still holds the request answers with the same binding; one that lost it creates the review again. A result the old binding delivers later is recorded as unmatched. Resubmitting relies on the authority still honouring that idempotency key. Before you resubmit a review coded `result-poll-attempts-exhausted` or `operator-closed`, confirm at the authority that it no longer holds the review: an authority whose idempotency retention has lapsed opens a second review while a reviewer may still be working the first. `close` accepts an accepted review without a result. It marks the submission failed with `operator-closed` and keeps the binding for reconciliation, and a result the authority delivers for it afterwards is refused. It does not contact the authority, so cancel the review there as well if it still exists. Both refuse a withdrawn proposal, a review whose result is already recorded, and any other state with `review_recovery.submission.ineligible`, whose message names the reason, state, and code. `resubmit` also refuses once request retention erased the review request (`request-erased`), or when the request no longer awaits review for that proposal version (`proposal-not-submitted`). Each operation writes an audit `request` entry to the `bregctl` companion audit file before it changes the submission, and its `response` after the change commits. An audit destination that refuses the request entry stops the operation before it changes anything. One that refuses the response after the commit is reported as `review_recovery.recovery.unaudited`: the recovery took effect, so read the submission's state before retrying. Retain the project sources, module locks, generated files, test receipt, and runtime file for each activation. Store secrets separately. {/* Evidence: crates/registry-bregctl/src/apply_lifecycle.rs; crates/registry-breg/src/review_recovery.rs, ReviewRecoveryOperatorService; crates/registry-breg/tests/postgres_change_requests.rs, an_operator_resubmits_or_closes_a_review_its_authority_lost; crates/registry-bregctl/src/review_recovery.rs; crates/registry-breg/src/review_store.rs, read_projection(); crates/registry-bregctl/src/doctor.rs, startup_diagnostic(); crates/registry-bregctl/src/lib.rs, VerifyArgs, verify(), write_doctor_success(), doctor_success_output_is_stable_in_human_and_machine_formats(), and doctor_reports_postgres_advisories_after_the_passing_checks_without_failing(); crates/registry-breg/src/postgres/baseline.rs, a_healthy_server_reports_only_the_connection_numbers(); crates/registry-breg/src/api/mod.rs; crates/registry-breg/src/startup.rs. The doctor output above was captured on 2026-09-25 from a local bregctl dev session (products/breg/DEV.md), running `bregctl doctor` in both formats against the session's runtime file with SSL_CERT_FILE naming the session's PostgreSQL CA. */} ## Turn encryption on over existing plaintext A successor package can turn `encrypted: true` on a field the registry already holds in plaintext. The apply seals the stored values through the engine-executed backfill the package's reviewed migration declares, and the history choice the project authored in that migration decides what happens to the plaintext the database retains. The two choices and their trade-offs are in [Field encryption for restricted fields](../../explanation/breg-field-encryption/); this section is the operator path around the apply, which [Change an active registry](../breg-changes/) prepares and applies. Run the preflight before you apply, while the runtime file still names the active package: ```sh bregctl field-encryption preflight \ --runtime-config /etc/breg/runtime.yaml \ --package /srv/registry/build-2/package ``` The preflight is read-only and value-free. It binds the same predecessor package an apply binds, refuses when the active revision is anything else, and reports counts per covered field: the live rows still carrying plaintext, the retained journal revisions that carry the field member, and the retained change-request snapshots, cached idempotency responses, and event payloads that mention the field. When a covered field declares a unique blind index, the preflight refuses when two of those plaintext values would normalize onto one index entry, naming the authored record identifiers, capped at 64, and never a value. Correct one of the named records and run the preflight again; the apply refuses the same collision before it seals anything. :::danger[Erase-and-rebaseline destroys retained record history] When the authored history choice is `erase-and-rebaseline`, the retained history of every record still carrying pre-flip plaintext is erased and snapshot coverage is restored with one rebaseline. Erasure is whole-record, so retained history beyond the encrypted field is destroyed with it, and nothing restores it. The erasure does not run as part of the apply: it waits until the flip's package is active, and you run it. Backups taken before the flip stay plaintext until you retire or re-encrypt them yourself. When the choice is `retain-plaintext-history`, nothing is erased and pre-flip revision snapshots keep serving as written, so the database-read guarantee does not hold for them. ::: After the successor package is active, write an owner-only JSON request file (mode `0600`) carrying the operator reference and the reason alone: ```json { "operatorReference": "approved-maintenance-003", "reason": "flip declared erase-and-rebaseline" } ``` Then run the erase-history lifecycle: ```sh bregctl field-encryption erase-history \ --runtime-config /etc/breg/runtime.yaml \ --request-file /srv/registry/private/erase-history.json ``` The request file names no record, entity, or field: the scope is the recorded `erase-and-rebaseline` flips themselves, so a document that names records or fields is refused rather than reinterpreted, and a request file that is not owner-only is refused before any connection is opened. For every record still holding pre-flip plaintext, the lifecycle erases its retained history through the same path [`history erase`](../breg-retention/#erase-retained-history) uses and restores snapshot coverage with one rebaseline, then writes one audit record that carries counts, not values. That rebaseline blocks reads and writes of every entity until it commits, so run the lifecycle in the maintenance window the [rebaseline outage guidance](../breg-retention/#restore-snapshot-coverage-after-an-erasure) describes. `bregctl field-encryption keygen --output ` writes one fresh base64 data key for the local file provider with owner-only permissions, never prints it, and refuses to overwrite an existing file. {/* Evidence: crates/registry-bregctl/src/lib.rs, FieldEncryptionCommand, FieldEncryptionPreflightArgs, FieldEncryptionEraseHistoryArgs, and FieldEncryptionKeygenArgs; crates/registry-bregctl/src/field_encryption_lifecycle.rs, run_preflight(), run_erase_history(), RawFieldEncryptionEraseRequest, and read_owner_only_request_file(); crates/registry-breg/src/field_encryption_backfill.rs, FieldEncryptionBackfillPreflightReport, MAX_NAMED_DUPLICATE_RECORDS, and pending_erase_targets(); crates/registry-breg/src/postgres/interlock.rs, field_encryption_duplicate_preflight(); crates/registry-breg/src/migration_plan.rs, ReviewedFieldEncryptionHistory; crates/registry-bregctl/src/field_encryption.rs, keygen(). */} ## Serve the container image The container serves only a package that activation already recorded, and the runtime document you activated with on the operator host is not the one the container serves. Write a second runtime document for the container, changing exactly four things from the host's: `listener.bind`, `secretProviders.file.root`, `package.root`, and the database hostname the resolved connection URL names, because the container mounts its own paths and reaches the database over its own network. ```yaml listener: bind: 0.0.0.0:8080 secretProviders: file: root: /etc/breg/secrets package: root: /var/lib/breg/package ``` Nothing restricts `listener.bind` to a loopback or private address the way `metricsListener.bind` is restricted, so the container can bind `0.0.0.0` and accept traffic from outside its network namespace; a host process that only ever serves `127.0.0.1` has no reason to. Mount the three configuration inputs read-only, add a writable audit volume owned by the container's non-root user, and publish the listener port. Set `BREG_IMAGE` to the release image digest from [Obtain the runtime](#obtain-the-runtime) before running: ```sh docker run --rm \ -p 8080:8080 \ -v /srv/registry/container/runtime.yaml:/etc/breg/runtime.yaml:ro \ -v /var/lib/breg/build-1/package:/var/lib/breg/package:ro \ -v /etc/breg/container-secrets:/etc/breg/secrets:ro \ -v /var/lib/breg/audit:/var/lib/breg/audit:rw \ "$BREG_IMAGE" ``` Run `bregctl verify` and `bregctl doctor` against this exact runtime document before you rely on the container's `GET /ready`, either from a host that reaches the same database and the same mounted package, or from the image as the next section shows. {/* Evidence: crates/registry-breg/src/runtime_config.rs, ListenerConfig and MetricsListenerConfig; release/docker/Dockerfile.breg. */} ### Activate from the image The image carries the `bregctl` built from the same source as its `breg`, so you can plan, apply, and read the activation ledger from the image digest you serve, without installing `bregctl` on an operator host. Override the entrypoint with `/usr/local/bin/bregctl` and pass the same arguments [Activate and serve](#activate-and-serve) uses, with the container's paths: ```sh docker run --rm \ --entrypoint /usr/local/bin/bregctl \ -v /srv/registry/container/runtime.yaml:/etc/breg/runtime.yaml:ro \ -v /var/lib/breg/build-1/package:/var/lib/breg/package:ro \ -v /etc/breg/container-migration-secrets:/etc/breg/secrets:ro \ -v /var/lib/breg/audit:/var/lib/breg/audit:rw \ "$BREG_IMAGE" \ --format json plan \ --runtime-config /etc/breg/runtime.yaml \ --package /var/lib/breg/package ``` Run `apply` the same way, replacing the last three lines with `--format json apply --runtime-config /etc/breg/runtime.yaml --package /var/lib/breg/package`, then read the ledger with `--format json status --runtime-config /etc/breg/runtime.yaml`. Add `--initial` to `apply` only when `plan` reports `activation: initial`, for the first package in an uninitialized database; a later package is applied without it. `plan`, `apply`, and `status` connect with the migration credential that `database.migrationUrlRef` names, so mount a secret directory that holds it beside the files the runtime document references, and keep that credential out of the serving container's secret directory. Make the image's non-root user, UID 65532, the owner of every secret file, as for the serving container, because the resolver refuses a file owned by anyone else. `apply` appends its audit entries to a file of its own beside `audit.path`, so the audit volume is mounted writable here too. A `--backup` binding file is read inside the container: mount it read-only and name its container path. In Kubernetes, run the same command as a Job whose container sets `command: ["/usr/local/bin/bregctl"]` and puts the subcommand and its flags in `args`. {/* Evidence: release/docker/Dockerfile.breg, the bregctl install line and the breg ENTRYPOINT; release/scripts/check-debian13-images.py, HTTP_PROBE_DOCKERFILES and check_repository(); crates/registry-breg/src/audit.rs, RegistryAudit::open_companion(); crates/registry-bregctl/src/apply_lifecycle.rs; crates/registry-bregctl/src/lib.rs, the plan, apply, and status argument definitions. */} ## Scrape metrics The listener that serves the registry API never serves metrics. To expose them, add a second listener that only the operator's network can reach: ```yaml metricsListener: bind: 127.0.0.1:9464 ``` The address must be a loopback or private address (an IPv4 private range, or IPv6 unique-local) with a non-zero port, and it must differ from `listener.bind`; the runtime file is otherwise refused with `runtime_config.invalid_metrics_listener`. The listener answers `GET /metrics` in the Prometheus text format and nothing else, and it carries no authentication, so keep it behind the scraper's network boundary rather than a public one. | Series | Type | Labels | | --- | --- | --- | | `breg_http_requests_total` | counter | `route`, `method`, `status` | | `breg_http_request_duration_seconds` | histogram | `route`, `method`, `status` | | `breg_anonymous_refusals_total` | counter | `route`, `method`, `reason` | | `breg_pool_connections` | gauge | `state`: `max_size`, `size`, `available`, `waiting` | `route` is the matched route template, such as `/v1/records/placements/{record_id}`, or `unmatched`; it never carries a record identifier. `status` is `success`, `client_error`, or `server_error` rather than the exact code. The label sets are closed, so a scrape cannot grow without bound, and no label carries a principal, a token, or a record value. Pool states are read from the connection pool at scrape time. `breg_anonymous_refusals_total` counts requests that presented no credential and were refused before admission: a profile the caller cannot hold, a route it cannot see, an absent scope or purpose, or a query the route cannot parse. `reason` is one of nine fixed values, `read_request_invalid`, `read_concealed`, `read_refused`, `revision_request_invalid`, `revision_concealed`, `revision_refused`, `mutation_concealed`, `mutation_refused`, and `action_refused`, derived from the refusal itself rather than from anything the request carried. These refusals are counted rather than journaled, because a caller with no principal names nobody the journal could hold accountable; refusals of an authenticated principal are journaled, and so is every admitted request. Alert on a sharp rise in this counter the way you would on a rise in `client_error` responses: it is the signal the journal no longer carries. An admitted anonymous read writes the same pre-I/O audit envelope as any admitted request, and the runtime enforces no request-rate limit of its own, so put a registry that admits anonymous reads behind an upstream rate limit. {/* Evidence: crates/registry-breg/src/metrics.rs, AnonymousRefusalReason; crates/registry-breg/src/api/mod.rs, anonymous_refusal(); crates/registry-breg/src/postgres/read.rs, record_pre_io_audit() call in execute(); crates/registry-breg/tests/postgres_anonymous_refusals.rs; crates/registry-breg/src/runtime_config.rs, MetricsListenerConfig; crates/registry-breg/src/startup.rs. */} ## Troubleshooting | Symptom | Next move | | --- | --- | | `test` refuses the runtime file | Compare it with the serving file: same identity, schema-test database references, and an empty package root. | | A secret reference is refused | The file under the secret root must be a regular file owned by the running user, with mode `0400` or `0600`, one link, and no symbolic link, and its name must follow the naming rule in [Create the secret files](#create-the-secret-files). | | `package` refuses the receipt | The receipt binds sources, baseline, and fingerprint. Rerun `test` for the exact candidate. | | `doctor` reports `startup.instance_claim.mismatch` | The database is a copy the registry's instance claim does not name, as after a logical restore. Adopt it once the original is stopped for good, as [back up and restore the database](../breg-changes/#back-up-and-restore-the-database) describes. | | `doctor` reports `startup.database.uninitialized` | No package was ever activated in this database. Run `plan`, then `apply --initial`, as [Activate and serve](#activate-and-serve) shows. | | `doctor` reports `startup.database.pre_ledger` | A release before the activation ledger activated this database. Rebuild the active project unchanged with this release's `bregctl package`, point `package.root` at the rebuilt package, then `plan` and `apply` it once with `--package` naming the same directory and without `--initial`; that apply adopts the database into the ledger. | | `doctor` reports `startup.database.identity_mismatch` | The database records another database id than `identity.databaseId`. Point the database URLs at the database the file names, or correct `identity.databaseId`. | | `doctor` reports `startup.package.not_active` | The ledger names another package as active than the one at `package.root`. Run `bregctl status` to see the active digest, then set `package.root` to that package, or `plan` and `apply` the package you meant to serve. | | `doctor` reports `startup.runtime_role.can_write` | In split role mode the runtime role can write the activation ledger or the registry state, for example through a grant, an ownership, a membership, or a trigger no compiled migration creates. Run `bregctl plan --package DIR`: its refusal names the object and the statement that removes it, such as `DROP TRIGGER`. | | `doctor` reports `startup.runtime_role.grants_missing` | The runtime role lacks a grant the active package gives it, as after `REASSIGN OWNED`. Run `bregctl apply --package DIR` with the active package to reissue the grants. | | `doctor` reports `startup.role_mode.changed` | The ledger records a split role activation, but the runtime file names one role for both. Restore the separate runtime role, or `plan` and `apply` the active package under one role, which records a `role_change` activation. | | `plan` or `apply` reports `apply.successor.roles_differ` | The runtime file names other database roles than the active activation serves with, and only a role change retires the recorded runtime role. Run `plan` and `apply` with the active package under the new roles, which records a `role_change` activation, then apply the successor. | | `doctor` reports `startup.oidc.refused` | Check the issuer, discovery reachability or the static JWKS document, the algorithm, and the leeway bound. | | `doctor` reports `startup.package.refused` with `the package compiler derivation failed` after an engine upgrade | The upgraded compiler derives a different schema from the active package, for example a new reference index. Build and apply a successor package with the upgraded `bregctl` before starting the upgraded `breg`, as [Change an active registry](../breg-changes/#test-and-package-the-successor) describes. `diff --runtime-config` reads the active package, and `test` and `package` accept its directory through `--baseline-package`; `verify` refuses it until the successor is active. | | `doctor` or startup warns `postgres.connections.pool_over_half` | Lower `database.pool.maxSize` or raise `max_connections` until every replica, operator command, and monitoring session fits; see [Set a PostgreSQL baseline](#set-a-postgresql-baseline). | | `doctor` or startup warns `postgres.autovacuum.off` or `postgres.track_counts.off` | Turn the setting back on in the server configuration and reload PostgreSQL, then run `ANALYZE` in the registry database so planner statistics catch up. | | The runtime file is refused with `runtime_config.invalid_metrics_listener` | `metricsListener.bind` must be a loopback or private address with a non-zero port, distinct from `listener.bind`. | | Every token is refused | Compare the token's header and claims with the token table; a list-valued `aud` or a missing principal claim refuses the token. | | An operator command cannot reach PostgreSQL | Check the migration URL secret, the role names, and `SSL_CERT_FILE` for a private authority. | | A command refuses a path that exists | Some component of the path is a symbolic link. Name the real directory; on macOS pass `/private/tmp` rather than `/tmp`. | {/* Evidence: crates/registry-breg/src/package.rs, rederive(), inspect_package_integrity(), and load_predecessor_package(); crates/registry-bregctl/src/package_inspection.rs, inspect_runtime_package() and inspect_baseline_package(); crates/registry-bregctl/src/doctor.rs, startup_diagnostic(); crates/registry-breg/tests/postgres_reference_indexes.rs, a_successor_adds_reference_indexes_to_a_database_activated_without_them. */} ## Next - [Bind webhook receivers](../breg-webhooks/) when the project declares events, because the package refuses to activate until every destination is bound. - [Change an active registry](../breg-changes/) for the successor package, its diff, and its migration evidence. - [Retain, erase, and audit](../breg-retention/) for the maintenance commands that erase history, erase expired Evidence uses, and retain the audit stream. - [Harden a production deployment](../../security/hardening-checklist/) for the controls around the listener, the secret root, and the migration credential. - [Evidence deployment targets](https://github.com/registrystack/registry-stack/tree/main/products/evidence/reference/deployment-targets) and the [Evidence Compose adapter](https://github.com/registrystack/registry-stack/blob/main/docker/compose/docker-compose.yaml) cover Evidence deployment shapes, not Base Registry Engine; read them when this deployment also runs Evidence alongside the registry.