Build a resumable data transfer

View full size

Data transfer and reconciliation are one governed lifecycle in Airlift: moving bytes is not complete until independently inspectable evidence satisfies the planned policy.

Airlift turns data movement into an executable, observable migration stage. It does not copy bytes itself. Your transfer runner—Lakeflow, a source-native unload/load job, or another certified implementation—moves data. Airlift binds that runner to accepted migration scope, starts a durable Temporal workflow, records every restart checkpoint through Fabric Platform, and refuses to complete when counts, lag, rejects, or evidence violate policy.

This separation gives a developer two useful guarantees:

1. retrying a worker does not create a second logical transfer; and
2. a successful copy job is not treated as proof of reconciled data.

Lifecycle

completed is system-derived. A runner submits immutable reconciliation evidence; Airlift checks the latest row counts, rejects, lag, restart reference, and configured threshold before projecting the terminal state.

Prerequisites

Before planning a transfer, create these governed records:

• an active or frozen engagement with an accepted assessment;
• in-scope data objects;
• a verified source binding with snapshot_read or change_feed_read;
• a verified target binding with target_write; and
• an immutable transfer_specification artifact.

Connection records contain opaque secret or Databricks connection references. Do not put credentials in a JSON file or action payload.

1. Register a transfer specification

{
  "engagementId": "eng_01J...",
  "estateId": "est_01J...",
  "kind": "transfer_specification",
  "name": "Orders snapshot and incremental catch-up",
  "artifactRef": {
    "system": "artifact_store",
    "type": "transfer_specification",
    "id": "orders-transfer-v3",
    "digest": "<64-character-sha256>"
  },
  "digest": "<64-character-sha256>",
  "mediaType": "application/json",
  "toolVersion": "migration-assets@3.0.0"
}

fa artifact register \
  --file transfer-spec-artifact.json \
  --idempotency-key orders-transfer-spec-v3

The artifact body stays in your artifact store. Airlift records its reference, digest, media type, producer generation, and scope.

2. Plan the transfer

{
  "engagementId": "eng_01J...",
  "estateId": "est_01J...",
  "waveId": "wav_01J...",
  "mode": "incremental",
  "sourceBindingId": "bnd_01J...",
  "targetBindingId": "bnd_01J...",
  "specificationArtifactId": "art_01J...",
  "profileId": "airlift.snapshot_incremental_lakeflow.v1",
  "toolVersion": "client-transfer-runner@1.4.2",
  "objectIds": ["obj_01J...", "obj_01K..."],
  "maximumLagSeconds": 300
}

fa transfer plan \
  --file transfer-plan.json \
  --idempotency-key orders-transfer-plan-v1

The first executable profile is airlift.snapshot_incremental_lakeflow.v1. The schema also represents snapshot, CDC, and streaming transfers, but each source/profile pair needs an admitted runner and live certification before a client engagement uses it.

3. Authorize and run

fa transfer run xfr_01J... \
  --idempotency-key orders-transfer-run-v1

This command authorizes canonical running state through airlift.transfer_start. In a deployed Airlift composition, the worker starts or attaches to transferWorkflowV1 with workflow ID airlift-v1-xfr-<transfer-id>. Duplicate starts attach to the same Temporal execution.

The runner receives opaque connection references and stable per-object/per-step idempotency keys. It returns snapshot, catch-up, restart, lag, and reconciliation references; it never writes Airlift projections directly.

4. Inspect and control the run

fa transfer status xfr_01J... --json
fa transfer pause xfr_01J... \
  --reason "Pause before source maintenance" \
  --idempotency-key orders-pause-maintenance
fa transfer resume xfr_01J... \
  --idempotency-key orders-resume-maintenance

Pause and resume are natural-person controls. A payload-free Temporal signal may wake the workflow after the governed action changes state, but it cannot carry identity or authority.

A checkpoint contains:

type TransferCheckpoint = {
  sequence: number;
  phase: 'snapshot' | 'catch_up' | 'streaming' | 'freeze';
  checkpointRef: ForeignReference;
  checkpointDigest: string;
  sourceWatermark: string;
  targetWatermark?: string;
  rowsRead: number;
  rowsWritten: number;
  bytesRead: number;
  bytesWritten: number;
  rejects: number;
  lagSeconds?: number;
  restartRef: ForeignReference;
  observedAt: string;
};

Only an admitted runner can record checkpoints. Sequences must increase exactly, and the workflow resumes at the first object without a durable checkpoint.

5. Reconcile

fa transfer reconcile xfr_01J... \
  --idempotency-key orders-reconcile-start

# Normally submitted by the admitted runner, not a person:
fa transfer reconcile-record \
  --file reconciliation-result.json \
  --idempotency-key orders-reconcile-result-v1

For a passing result, Airlift requires at least one checkpoint, equal source/target row counts, zero rejects, and lag at or below maximumLagSeconds. The runner's passed claim cannot override these checks.

Failure and uncertainty

Use the Data transfer screen in the Databricks App to inspect active runs, checkpoint counts, lag thresholds, row counts, restart references, runner generations, and reconciliation state.