Move data in bulk

Use bregctl data to load many records from a file into a running registry, or to copy many records out of it. Every row goes through the same authenticated routes, grants, validation, and audit as a request from an application. An import commits one chunk at a time and an export appends one page at a time, each writing a checkpoint, so you can stop a run and continue it later.

The examples load operators into the facility registry that author a registry project builds, deployed as deploy a registry describes.

Prepare the inputs

You need three things:

• The package directory you activated, for example /srv/registry/build-1/package.
• The registry’s URL. It must use https, unless the host is loopback.
• A file holding one bearer token and nothing else, issued to an access profile that may write batches of the Record type. A trailing newline is fine; any other whitespace refuses the file.

The Record type must declare batch bounds, maximumItems (up to 100) and maximumBytes for one chunk, and the profile’s grant on it must include batch. The configuration reference lists both keys.

Each input line is one JSON object naming the operation and the record’s fields:

operators.jsonl

{"operation": "create", "data": {"operatorCode": "OP-101", "status": "active", "jurisdiction": "zone-a"}}
{"operation": "create", "data": {"operatorCode": "OP-102", "status": "active", "jurisdiction": "zone-a"}}
{"operation": "create", "data": {"operatorCode": "OP-103", "status": "active", "jurisdiction": "zone-a"}}
{"operation": "create", "data": {"operatorCode": "OP-104", "status": "inactive", "jurisdiction": "zone-a"}}
{"operation": "create", "data": {"operatorCode": "OP-105", "status": "active", "jurisdiction": "zone-a"}}

A file holds at most 1,000,000 lines and 256 MiB, and a patch line at most 128 operations. An empty file, a symbolic link, or a file over a bound is refused before any request. Every path you pass to bregctl data must be absolute.

Validate the input

data validate checks every line against the package without touching the network, so a shape error surfaces before a single request:

bregctl data validate \
  --package /srv/registry/build-1/package \
  --record-type operator --profile operator-editor --operation create \
  --input /srv/registry/import/operators.jsonl

Validated the input. 5 items in 3 chunks.

The report goes on to name the package revision, the profile and operation, and the file’s input sha256, which the import window below can pin. A line that does not fit the Record type or the profile’s grants refuses the file with data.validate.input.refused, without echoing record values.

Import with a checkpoint

data import commits the file as an ingestion run on the server, one chunk at a time. The checkpoint file, and the .state file written beside it, name that run. --max-chunks stops after that many chunks, so you can watch the first ones land:

bregctl data import \
  --package /srv/registry/build-1/package \
  --breg-url https://registry.example.org \
  --access-token-file /srv/registry/private/import-token \
  --record-type operator --profile operator-editor --operation create \
  --input /srv/registry/import/operators.jsonl \
  --checkpoint /srv/registry/import/operators.checkpoint --max-chunks 1

Imported the input. 2 items committed, and the import is not complete.

Run the same command again, with or without --max-chunks, to continue. When the last chunk commits, the report says so:

Imported the input. 5 items committed.

A rerun asks the server where the run stands and sends only the chunks after that point, even when the local checkpoint lags behind. A chunk whose answer was lost in transit is sent again with the same bytes, and the server returns its original receipt instead of writing twice. Running the command once more after the import is complete writes nothing.

Leave the checkpoint and its .state file as they are. They bind the package, Record type, profile, operation, and run, so a rerun with any of those changed is refused rather than resumed.

Caution

Importing lines that already committed again, under a new checkpoint path, creates their records a second time.

Load records through an import window

When a Record type’s mutationPolicy sets create to proposal-required, the registry refuses direct creates and batches: each new record needs a Proposal. For an initial load, give the loading profile an import grant instead of batch. An import grant only creates records, only through ingestion runs, and only while an operator holds an import authority open: a window bounded by Record type, profile, item count, and expiry. The commands below use the facility template’s directory-entry Record type and directory-editor profile, set up that way.

Without an open window, the import is refused and no run is created:

bregctl data import refused.

  error    data.import.ingestion-run.import-authority-required  ingestionRun

The window is opened over the migration connection in the runtime configuration, which no API caller holds. Validate the file first, then open a window sized to it and pinned to the digest data validate reported:

bregctl import-authority open \
  --runtime-config /etc/breg/runtime.yaml \
  --record-type directory-entry --profile directory-editor \
  --max-items 5 --expires-in 2d \
  --input-sha256 <input sha256 from data validate> \
  --operator-reference change-1482 --reason "Directory initial load"

Opened the import authority.

  authority id                     <authority-id>
  status                           open

--expires-in takes whole minutes, hours, or days (90m, 12h, 7d); the default is 7d, the most is 30d, and a window cannot be extended. --input-sha256 may repeat up to 16 times; without it, any file within the item count is admitted. The digest is a label the client computes and announces; the server never sees the file, so the item count is the bound it enforces. The operator reference and reason are stored only as keyed hashes, so you cannot read them back.

Now run data import with the directory-editor profile, as in the previous section. The window ends when its committed items reach the count, which list shows as exhausted:

bregctl import-authority list --runtime-config /etc/breg/runtime.yaml

Listed the newest import authorities. 1 authority.

  authority id                     <authority-id>
  status                           exhausted

A Record type holds at most one open window. A window also ends when you close it with bregctl import-authority close --authority-id <authority-id>, when it expires, or when another package is activated (superseded). Run bregctl import-authority close-expired on a schedule to record the expiries no load has noticed yet. Every opening and ending appends one record to the audit journal.

Each chunk rechecks the window. When the window ends mid-load, the committed chunks stay and the run is blocked. A blocked run cannot continue: open a new window and import only the lines the run did not commit, under a new checkpoint path.

Resume a load through a durable ingestion run

The checkpoint lives beside the client, but the run lives on the server. If the machine that started a load is gone, or you are not sure whether a file already loaded, ask the server before importing again. List the runs for the file’s digest, with a header file holding the Authorization: Bearer line:

curl -s -H @/srv/registry/private/import.header \
  "https://registry.example.org/v1/records/operators/ingestion-runs?inputDigest=<input sha256>"

Each run in runs reports its status, itemCount, committedItems, and nextChunkIndex, but never the rows themselves. The list also filters by status, and pages with cursor and nextCursor. The status tells you the next move:

Status	Next move
open	Rerun data import with the original checkpoint, or continue the run through its routes.
complete	The file is loaded. Do not import it again.
blocked	Read blockedReason, then import the lines after committedItems under a new checkpoint.
cancelled	Someone stopped the load on purpose. Its counts and audit are kept.

A run is blocked with activePackageChanged when another package is activated, and with importAuthorityClosed when its window ends. To continue a run from another machine without its checkpoint, drive its routes directly, as the ingestion-run reference describes.

Export with a checkpoint

data export pages through the list route and appends each page to the output file, then records the output length, digest, record count, and server cursor in its checkpoint. Name each field with --field, using its API name. The profile must have allowDataExport on the Record type, and bregctl check warns when it does, because data already downloaded cannot be recalled:

bregctl data export \
  --package /srv/registry/build-1/package \
  --breg-url https://registry.example.org \
  --access-token-file /srv/registry/private/export-token \
  --record-type operator --profile operator-editor \
  --field operatorCode --field status \
  --output /srv/registry/export/operators.jsonl \
  --checkpoint /srv/registry/export/operators.checkpoint

Exported the records. 5 records.

Each output line is one record with its id, revision, and the fields you named. --max-pages bounds one run the way --max-chunks bounds an import; run the same command again to continue.

The first run creates both files, and refuses to start when only one of them exists. A rerun checks the output against its checkpoint, length and digest included, before it asks for the next page, so an output edited by hand is refused rather than extended. A run stopped between appending a page and recording it leaves one unrecorded page, which the rerun discards and fetches again. Keep each output with its checkpoint, and give every export its own pair.

When a command refuses with a diagnostic code, Monitor and recover gives the next move for each one.

Next

• Base Registry Engine API reference for the batch, list, and ingestion-run routes these commands drive.
• Control access and reads for the grants an import or export must hold.
• Retain, erase, and audit for the audit journal every imported row is recorded in.