Move data in bulk

You operate an active registry, deployed as deploy a registry describes, and need to load many records from a file or copy many records out through the same authenticated routes an application uses. At the end of this page an import has committed every chunk and its report says it is complete, or an export file holds every page the profile may read, and either run can be interrupted and resumed from its checkpoint.

data import and data export drive the ordinary authenticated routes, so every row passes the same grants, validation, and audit as an interactive client: the import commits its chunks through the entity’s ingestion-run routes, and the export pages through the list route. Nothing here bypasses a profile: an import needs a profile that may create or patch the entity, and an export needs one that is export-enabled. A chunk is one batch request the import commits as a unit; a page is one list response the export appends as a unit; a checkpoint is the file each command writes after every unit so a rerun continues where the last one stopped.

Prepare the inputs

Each command needs the activated package directory and, for the two networked commands, the server URL and a file holding one bearer token and nothing else. The token file is an absolute path to a regular file; a trailing newline is ignored, and any other whitespace or control character refuses it. --breg-url must be https; http is accepted only for a loopback host. Each import line is one JSON object naming the operation and the record data:

{"operation": "create", "data": {"code": "AA", "label": "First"}}

The input is a regular file of at most 256 MiB holding at most 1,000,000 lines, and a patch line carries at most 128 operations. An empty file, a symbolic link, or a file over either bound is refused before any network use. Paths given to --package, --input, --checkpoint, and --output must be absolute.

Validate the input

data validate checks every line against the compiled plan without a network, so a shape error surfaces before a single request is sent:

bregctl data validate \
  --package /srv/registry/build-1/package \
  --entity record --profile operator --operation create \
  --input /srv/registry/import/records.jsonl

The report names the package revision and schema fingerprint it validated against, the entity, profile, and operation, the input length, and the item and chunk counts the import would use. A line that does not fit the entity’s shape or the profile’s grants is reported without record values.

Import with a checkpoint

An import commits one bounded chunk at a time as a durable ingestion run: the server holds the committed boundary and the receipt of every chunk, while the checkpoint file beside its .state sidecar names the run the command created, so a rerun with the same checkpoint resumes the same run after the last committed chunk and never re-sends a committed one. --max-chunks bounds one operator run, which lets you watch the first chunks land before committing the rest:

bregctl data import \
  --package /srv/registry/build-1/package \
  --breg-url https://registry.example.org \
  --access-token-file /srv/registry/private/import-token \
  --entity record --profile operator --operation create \
  --input /srv/registry/import/records.jsonl \
  --checkpoint /srv/registry/import/records.checkpoint --max-chunks 20

Run the same command again to continue. The report carries the ingestion run id, the number of completed chunks, the number of committed items, and whether the import is complete; a run that stops at --max-chunks reports it as incomplete, which is the signal to run again. The sidecar binds the package revision, schema fingerprint, entity, profile, operation, and run, so a rerun with any of them changed is refused rather than resumed. On a rerun the server is authority over progress: the command re-reads the run it names and continues from the server’s nextChunkIndex even when the local checkpoint lags or leads it, and a chunk answer lost in transit is recovered by re-reading the run and submitting the exact same chunk bytes, which replays the original receipt instead of writing again. Do not edit the checkpoint or the sidecar: a checkpoint without its sidecar, or one whose contents no longer match, is refused, and --max-chunks 0 is refused as an invalid binding. A sidecar written before ingestion runs (apiVersion v1) is refused rather than upgraded, because resuming its committed items under a new run id would duplicate mutations; a run that reports blocked or cancelled refuses new chunks and surfaces as its own error, and the run stays inspectable.

Resume a load through a durable ingestion run

The checkpoint of an import is a file beside the client, so the process that resumes must reach the file the process that started wrote. When that is wrong for your recovery story, drive the same chunking as a durable ingestion run over the API instead: the server holds the checkpoint, and any caller with the access token and the source file continues it. data import drives exactly this protocol, so its runs appear in the same listing and answer the same reads. A run stores no source rows; reading one returns operational metadata and bounded failure classifications only.

One run covers one input file. Create it on the entity’s ingestion-runs route: the create or patch operation, the profile, the package revision and schema fingerprint, the input length, and the item and chunk counts are what data validate reports for the same input; the run also needs the source digest and the chunking algorithm greedy-canonical-http-batch-v1. Submit each chunk in order, naming its index, its digest, and the digest of the source prefix it ends at. Every submission rechecks the profile against the entity’s batch route, so a run id alone grants nothing.

A run is open while chunks are due, complete when the last chunk commits, cancelled after an explicit cancel, and blocked once the active package no longer matches its binding. Each failure has one recovery:

Failure	Next move
A response is lost	Reread the run, then submit the same chunk again. An exact replay returns the original receipt and writes nothing new.
The transport drops a submission	Resubmit the chunk the run’s `nextChunkIndex` names.
A chunk holds an invalid item or a business refusal	The checkpoint stays where it is. Start a successor run for the remainder; skipping the refused rows is never a default recovery.
Authorization is lost	Progress refuses until the selected profile satisfies the batch route again.
The package or schema changed	The run reports `blocked` with reason `activePackageChanged` and stays inspectable. Start a successor run under the new binding.
You stop the load on purpose	Cancel the run. Its counts and audit are preserved.

Export with a checkpoint

An export appends one bounded page at a time to the output file, then records the output length, digest, record count, and server cursor in its checkpoint. --field names each field to export, and the profile must be export-enabled for the entity:

bregctl data export \
  --package /srv/registry/build-1/package \
  --breg-url https://registry.example.org \
  --access-token-file /srv/registry/private/export-token \
  --entity record --profile operator --field code --field label \
  --output /srv/registry/export/records.jsonl \
  --checkpoint /srv/registry/export/records.checkpoint --max-pages 50

The first run creates both files and refuses to start when either already exists on its own. A rerun streams the checkpointed prefix of the existing output, matches it against the checkpoint, length and digest included, and only then requests the next server-validated cursor, so an output edited by hand is refused rather than extended. A run stopped between the two writes leaves one page the checkpoint never recorded; the rerun discards that page and fetches it again. It refuses when the output is shorter than its checkpoint, when the checkpointed prefix no longer matches, when more than one page follows the checkpoint, or when anything follows a checkpoint that already reports the export complete. The report names the requested fields, the completed page count, the record count, the output length, and whether the export is complete. --max-pages bounds one run the way --max-chunks bounds an import, and --max-pages 0 is refused. Each server response is bounded at 2 MiB.

Troubleshooting

Symptom	Next move
`--breg-url` is refused	Use `https`, or `http` with a loopback host only.
The token file is refused	It must be an absolute path to a regular file holding one token and nothing else; strip everything but a trailing newline.
The input is refused before any request	Check for an empty file, a symbolic link, more than 256 MiB, more than 1,000,000 lines, or a patch line with more than 128 operations.
An import refuses its checkpoint	The sidecar is missing, or the package, entity, profile, or operation differs from the run that created it. Start a new checkpoint path for a new input.
An import reports `data.import.checkpoint.legacy`	The sidecar predates ingestion runs and names no run to resume; resending its committed items under a new run id would duplicate mutations. Start a new import with a fresh checkpoint path.
An import reports the run was cancelled	The named run was cancelled and refuses new chunks; its counts and audit are preserved. Start a new import with a fresh checkpoint path.
An export refuses to start	Exactly one of the output and checkpoint files exists, or the existing output no longer matches its checkpoint. Keep both files together and unedited, or start both afresh.
`data export` is refused with `data.export.checkpoint.refused`	The output file and its checkpoint no longer describe one another: the output is shorter than the checkpoint, its checkpointed prefix changed, more than one page follows the checkpoint, or the checkpoint belongs to another export. Keep both files and resume from copies you trust. A rerun discards at most the single page an interrupted run left unrecorded.
The report says the run is incomplete	`--max-chunks` or `--max-pages` bounded it. Run the same command again to continue.
A durable ingestion run reports `blocked`	The active package no longer matches the binding the run was created under. Read the run, then start a successor run under the new package; the blocked run stays inspectable. `data import` surfaces the same state as `data.import.ingestion_run.blocked`.

Base Registry Engine API reference for the batch and list routes these commands drive, the durable ingestion-run routes, and the problems they return.
Query a registry from Python and Node for the client library that reads the same routes one request at a time.
Control access per profile for the profile grants an import or export must hold.
Retain, erase, and audit for the audit journal every imported row is recorded in.