Skip to content
Registry StackDocsDevelopment (unreleased)

Bind webhook receivers

For the operator

View as Markdown

If your registry project declares events that name a webhook destination, bind each destination to a receiver, give the receiver the signing key, build it against a sample request, and recover deliveries that did not arrive. The examples use one event, contact-created-v1 on the contact Record type, sent to a destination called contact-events.

The project names a destination only by identifier, so the same package runs in any deployment; the runtime file binds that identifier to a real receiver. Base Registry Engine (BReg) sends an event after the write that caused it commits, through its outbox. Send registry events to a webhook shows a first delivery on a local registry.

Add an eventDestinations block to the runtime file, with one entry per destination the package names:

runtime.yaml
eventDestinations:
contact-events:
origin: https://receiver.example.org
path: /events
networkProfile: production-https
dnsFamily: dual-stack-strict
allowedPrivateCidrs: []
hmacSha256KeyRef: secret:file/contact-events-key
classificationCeiling: internal
deliveryCeilings:
attemptTimeoutMilliseconds: 1000
maximumAttempts: 3

bregctl check <project> --runtime-config runtime.yaml checks the binding’s shape offline. The runtime refuses to start when the bindings and the package disagree: every destination the package names needs a binding, and a binding for a destination the package does not name is refused.

Some values need a decision:

  • classificationCeiling must cover the classification of every field the event sends. A binding cannot add fields to a payload.
  • production-https refuses localhost and every name under it. Use loopback-development-http only for a receiver on a development host.
  • For a receiver on a private network, list its address ranges in allowedPrivateCidrs. Use dnsFamily: ipv4-only only on a network without IPv6.
  • For a receiver behind a private certificate authority, or one that requires mutual TLS, name the PEM material with tls.caBundleRef and tls.clientIdentityRef.
  • deliveryCeilings may tighten what the package compiled, never widen it. This example tightens both values.

bregctl --format json explain events <project> shows what the package compiled for each delivery: by default five attempts of five seconds each, retried after 1, 2, 4, and 8 seconds, then a dead letter. Retries keep the event id and idempotency key. Any 2xx response acknowledges a delivery; redirects, timeouts, and every other response are retried. The configuration reference lists every key, including retention.hookPayloadRetentionDays.

Run bregctl doctor after every change to this block. It opens each binding, key file included, before the runtime starts.

Generate the HMAC-SHA-256 key as an owner-only file in the runtime’s secret directory, as the user that runs breg:

Terminal window
(umask 077; openssl rand -hex 32 | tr -d '\n' > /var/lib/breg/secrets/contact-events-key)

Give the receiver the same bytes through its own secret store. BReg signs with the file’s bytes exactly as written, so the receiver loads the 64 characters as its key and does not decode them. The key must be at least 32 bytes, and the file a regular file owned by the user that runs breg with mode 0400 or 0600; doctor and startup refuse anything else.

bregctl webhook sample prints the request one event produces, without a runtime or a database, so you can build the receiver before the registry serves:

Terminal window
bregctl webhook sample ./my-registry --event contact-created-v1
Built the sample delivery. The canonical request follows.
event contact-created-v1
POST <configured-webhook-request-target> HTTP/1.1
Accept: application/json
Content-Type: application/json
Idempotency-Key: sha256:0000000000000000000000000000000000000000000000000000000000000000
X-Registry-Delivery-Attempt: 1
X-Registry-Delivery-Time: 2026-01-01T00:00:00Z
X-Registry-Event-Generation: 1
X-Registry-Signature: v1=<computed-at-delivery>
ce-dataschema: urn:breg:event-schema:local-directory:contact:contact-created-v1:sha256:644104a736a00be2962cf3682f869cc1a6d17b101a2d747927302dddb49d9f34
ce-id: 00000000-0000-4000-8000-000000000001
ce-source: urn:registrystack:registry:local-directory:instance:<configured-instance>
ce-specversion: 1.0
ce-time: 2026-01-01T00:00:00Z
ce-type: contact-created-v1
{"causation":{"hop":0,"root":"00000000-0000-4000-8000-000000000001"},"data":{"packageRevision":"sha256:0000000000000000000000000000000000000000000000000000000000000000","recordId":"00000000-0000-4000-8000-000000000002","recordType":"contact","revision":1,"trigger":"created","values":{"name":"x"}},"dataschema":"urn:breg:event-schema:local-directory:contact:contact-created-v1:sha256:644104a736a00be2962cf3682f869cc1a6d17b101a2d747927302dddb49d9f34","id":"00000000-0000-4000-8000-000000000001","source":"urn:registrystack:registry:local-directory:instance:<configured-instance>","subject":{"recordReference":"hmac-sha256:0000000000000000000000000000000000000000000000000000000000000002","recordRevision":1},"time":"2026-01-01T00:00:00Z","type":"contact-created-v1"}

The event type, data schema, registry id, and projected field names come from your project. bregctl fills in the rest: fixed placeholder identifiers, time, and signature, and a synthetic value for each projected field (here "name":"x"). --format json returns the same request as a document with the method, request target, headers, body, and canonical body.

The receiver should:

  1. Verify X-Registry-Signature against the exact request bytes, in the signing-input order the API reference defines.
  2. Check X-Registry-Delivery-Time for freshness.
  3. Read the Record type from data.recordType and the record from data.recordId, and order events for one record by data.revision.
  4. Deduplicate by Idempotency-Key, and answer 2xx only after it has durably recorded the event.

Activation checks local Rhai and Wasm deliveries across captured revisions. Pending or leased work and operator-replayable dead letters block activation while their payload is retained and unexpired. This applies to Record and Proposal hooks, even when the successor leaves the program unchanged. Other terminal metadata, including non-replayable dead letters, can remain, as can metadata whose payload expired or was erased. bregctl plan and apply report apply.hooks.local-retained before maintenance starts. Both name the retained count and compiled delivery identifiers, bounded to 100 deliveries. A truncated count means at least that many remain. Plan changes nothing; apply records a refused activation. The current package stays ready and no target is pinned, so this refusal needs no reconciliation.

If ongoing activity keeps replenishing the queue, you can choose a quiet window. Use your existing service or container supervisor to stop every breg process serving this registry database, keep all replicas stopped, and wait for process termination. This stops both HTTP serving and background producers; routing HTTP traffic away alone leaves background producers running. The registry is unavailable while these processes are stopped. The quiet window is optional: concurrent work remains retained, and apply rechecks the guard and refuses activation when local work remains.

Drain runs reviewed Rhai or Wasm handlers in the bregctl process using the runtime database connection. Use the active package’s runtime file, handler assets and Wasm execution configuration, with its configured destination secrets and audit destination and secrets available to that process. For encrypted fields, provide the existing field-encryption provider and keys. You can run the CLI on a different authorized operator host with access to those assets, secrets, and the runtime database. Keep the same secret ownership and field-key custody restrictions as the serving process, including restrictions on the local-only file key provider. Drain does not need a migration credential. Listing, replay and discard do not enable the execution-only Wasm runtime or field-encryption service.

Run the bounded drain:

Terminal window
bregctl --format json webhook drain \
--runtime-config /etc/breg/runtime.yaml \
--max-attempts 100 --timeout-seconds 30

The command attempts only due local work under the active package’s existing authority. That covers all the local work there is: activation refuses while local work that can still run is retained under any activation, so no earlier package’s local work survives a go-live. It stops when no local attempt is due, after the attempt budget, or at the time budget. completedAttempts counts settled attempts and stopReason explains the stop. remaining contains the bounded retained count, compiled delivery identifiers, and truncated flag. A time-budget stop can leave an unfinished lease for normal recovery. URL deliveries, live leases, retries that are not due, and dead letters remain for their existing worker or operator recovery; drain never replays or discards them. The attempt budget accepts 1 to 1000 and defaults to 100; the time budget accepts 1 to 300 seconds and defaults to 30. A drain can itself produce further governed hook work, so one bounded call does not guarantee an empty queue.

If work remains, inspect it with webhook list. Wait for a live lease or scheduled retry, or correct a runtime failure and explicitly replay eligible work. Replay uses the captured handler under the active package. If recovery requires changing governed handler code, assess and explicitly discard that retained work before activating the successor, as described in the recovery section. Run plan again before applying the successor. Apply checks the retained work again, because new work can arrive after a plan. If you chose a quiet window, keep the serving processes stopped through drain and apply. After successful activation, restart them on the successor as the activation procedure describes. If you defer cutover, restore service on the still-active package.

A URL receiver can acknowledge an event with 2xx after a compatible successor goes live. That acknowledgement does not mean the registry accepted the proposed change in its answer. An uncommitted proposal captured under the previous activation and first classified after go-live is dead-lettered with proposalCode: hook.proposal.package_unavailable. For this superseded-activation refusal, proposalSummary names the captured and active activation IDs, and replayEligible is false. Derive the change again from a fresh event; replay cannot give an old answer successor authority. An identical answer whose change already committed can still recover its original receipt without applying the change twice. A proposal classified before go-live keeps its applied, refused, or dead-lettered result when delivery completion succeeds afterward. The predecessor worker cannot claim new work.

If the worker is interrupted before recording the proposal result and there is no committed receipt to recover, final-lease recovery records worker-interrupted without a proposal code. Explicit replay can remain eligible when the captured operator permission, product proposal policy, active binding, and retained, unexpired payload allow it. A replay sends the event to the receiver again, and that is all it can do for a delivery captured before go-live. Any proposal the receiver returns is classified under the successor and dead-lettered as hook.proposal.package_unavailable, so the replay never applies a change. Inspect the recovered delivery before deciding whether to replay or discard it.

webhook list shows deliveries that are pending, leased, dead-lettered, or expired:

Terminal window
bregctl --format json webhook list --runtime-config /etc/breg/runtime.yaml --limit 50
{
"apiVersion": "id.registrystack.org/formats/breg/ctl-report/v1alpha1",
"kind": "BRegCtlReport",
"ok": true,
"command": "webhook list",
"deliveries": [
{
"eventId": "<event-id>",
"deliveryId": "events.contact.contact-created-v1.webhook",
"generation": 1,
"state": "dead-lettered",
"attempt": 5,
"payloadAvailable": true,
"payloadExpiresAt": "<expiry-time>",
"replayEligible": true,
"discardEligible": true,
"bindingActive": true,
"deadLetterReason": "http-non-success"
}
]
}

The list carries delivery metadata and no record values; --limit accepts 1 to 100. Delivered events are left out, so an empty list does not prove a delivery arrived: confirm that from the receiver’s own records.

A delivery that runs out of attempts becomes a dead letter with a deadLetterReason, and its payload is retained. replayEligible is true only when the captured policy permits replay, its exact binding is still deployed (bindingActive), the payload is retained, and the proposal outcome permits replay. Payloads are kept for retention.hookPayloadRetentionDays, 7 days by default and at most 30, and erased earlier once every delivery that needs them has arrived.

After you fix the receiver, replay the dead letter with the ids and generation from the list:

Terminal window
bregctl webhook replay --runtime-config /etc/breg/runtime.yaml \
--event-id <eventId> --delivery-id <deliveryId> --expected-generation <generation>
Queued the delivery for replay.
event id <event-id>
delivery id events.contact.contact-created-v1.webhook
generation 2

Replay keeps the event id and issues a new generation and idempotency key. If you are not sure whether the receiver acted on an earlier attempt, check its state before replaying. A replay whose generation is stale, whose payload has expired, or whose destination is no longer bound is refused with webhook.operation.refused; list again and use the current generation.

Discard deliveries pinned to an old binding

Section titled “Discard deliveries pinned to an old binding”

When you correct a binding while deliveries still wait under the old one, doctor reports startup.webhook.retained-bindings, and those deliveries show bindingActive: false. The runtime will not start until they are gone: restore the old binding until they drain, or discard each one whose discardEligible is true:

Terminal window
bregctl webhook discard --runtime-config /etc/breg/runtime.yaml \
--event-id <eventId> --delivery-id <deliveryId> --expected-generation <generation>
Discarded the delivery. It cannot be replayed.
event id <event-id>
delivery id events.contact.contact-created-v1.webhook
generation 3

The row stays in the list as expired, without its payload. A delivery whose lease is still live is refused: wait for the lease to expire and list again.

The webhook commands connect to PostgreSQL with the runtime file’s database settings. If the database presents a certificate from a private certificate authority, name its PEM bundle with database.trustedRootCertificateRef in the runtime file.

When doctor or a webhook command refuses, look up its diagnostic code in monitor and recover.