Build a conversion factory

View full size

Airlift does more than track a checklist. It gives your conversion worker a governed execution model: accepted scope becomes a batch; every object gets a reproducible attempt; every successful output gets an immutable artifact identity; every failure becomes visible remediation work.

The converter still does the translation. Lakebridge is the deterministic converter, and a bounded Harness agent may propose one repair candidate for admitted residue. Airlift coordinates those tools and prevents their success response from being mistaken for proof of migration readiness.

Converter success is an attempt outcome, not a readiness verdict. Airlift requires a matching immutable target artifact before a converted object can satisfy batch reconciliation, and it requires independent validation before certification.

Runtime model

accepted assessment
       │
       ▼
conversion batch ──pins── object IDs + method + tool generation
       │
       ├── conversion attempt ──success── immutable target artifact
       │                                  │
       │                                  └── independent validation
       │
       └── conversion attempt ──failure── residue case
                                          ├── bounded agent candidate
                                          └── human engineering

ConversionBatch is coordination state. It does not contain source code and it does not replace the object ledger. Artifact is an immutable reference plus a SHA-256 digest; the artifact body stays in your admitted repository, Unity Catalog volume, or release store. Conversion is one attempt against one object. Retrying creates another attempt, which preserves the history engineers need when comparing tool generations.

The CLI maps directly to Platform actions: batch create/start/complete invoke airlift.conversion_batch_create, airlift.conversion_batch_start, and airlift.conversion_batch_complete; attempt start/record invoke airlift.conversion_start and airlift.conversion_record; artifact registration invokes airlift.artifact_register.

1. Create a batch from accepted scope

Only objects from the accepted assessment are eligible. Objects must be planned or rework, and every object in a batch must belong to the same estate and assessment.

Create batch-create.json:

{
  "engagementId": "eng_01J00000000000000000000000",
  "assessmentId": "asm_01J00000000000000000000000",
  "name": "Stored code batch 1",
  "method": "lakebridge",
  "toolVersion": "lakebridge@YOUR_PINNED_GENERATION",
  "objectIds": [
    "obj_01J00000000000000000000001",
    "obj_01J00000000000000000000002"
  ]
}

fa conversion batch create \
  --file batch-create.json \
  --idempotency-key stored-code-batch-1

fa conversion batch list \
  --engagement-id eng_01J00000000000000000000000

Use lakebridge for the deterministic pass, agent only for an admitted bounded repair pass, and manual for engineer-authored conversion. The method is evidence; do not label human work as deterministic conversion.

2. Start the batch

Create batch-start.json:

{ "conversionBatchId": "cbh_01J00000000000000000000000" }

fa conversion batch start \
  --file batch-start.json \
  --idempotency-key stored-code-batch-1-start

Starting admits attempts for the pinned object set. conversion_start rejects an object, method, or tool generation that differs from the running batch. This prevents a worker configuration change from silently mixing outputs inside one batch.

3. Record one attempt per object

Create attempt-start.json:

{
  "objectId": "obj_01J00000000000000000000001",
  "batchId": "cbh_01J00000000000000000000000",
  "method": "lakebridge",
  "toolVersion": "lakebridge@YOUR_PINNED_GENERATION"
}

fa conversion attempt start \
  --file attempt-start.json \
  --idempotency-key stored-code-batch-1-object-1-start

After the adapter writes the output to your artifact store, create attempt-record.json:

{
  "conversionId": "cnv_01J00000000000000000000000",
  "objectId": "obj_01J00000000000000000000001",
  "outcome": "converted",
  "artifactRef": "volume://catalog/schema/migration/target/view.sql",
  "inputDigest": "INPUT_SHA256",
  "outputDigest": "OUTPUT_SHA256",
  "validationRunRef": "experiments-run-reference"
}

fa conversion attempt record \
  --file attempt-record.json \
  --idempotency-key stored-code-batch-1-object-1-result

fa conversion list --object-id obj_01J00000000000000000000001
fa conversion diff obj_01J00000000000000000000001 --json

convert diff returns the object's attempt history for machine comparison. Compare inputDigest, outputDigest, toolVersion, modelVersion, promptVersion, and validationRunRef; fetch code bodies from the referenced artifact store rather than from Airlift.

4. Register the immutable output

Converted attempts must have a matching target_code artifact before the batch can complete. Create artifact.json:

{
  "engagementId": "eng_01J00000000000000000000000",
  "objectId": "obj_01J00000000000000000000001",
  "kind": "target_code",
  "name": "Converted customer view",
  "artifactRef": {
    "system": "databricks",
    "type": "volume_object",
    "id": "catalog/schema/migration/target/view.sql",
    "digest": "OUTPUT_SHA256"
  },
  "digest": "OUTPUT_SHA256",
  "mediaType": "application/sql",
  "toolVersion": "lakebridge@YOUR_PINNED_GENERATION"
}

fa artifact register \
  --file artifact.json \
  --idempotency-key artifact-customer-view-v1

fa artifact list --object-id obj_01J00000000000000000000001

Register a new artifact with supersedesArtifactId when content changes. Never mutate a previous artifact row. Airlift rejects a reference digest that differs from the declared artifact digest.

5. Route failed attempts

Record a failed attempt with an actionable diagnostic. Then open a residue case before completing the batch:

The CLI invokes the governed airlift.residue_create action; it does not insert a work item or mutate the object projection directly.

fa residue create \
  --file residue-create.json \
  --idempotency-key object-2-residue-1

See Engineering remediation for the full state machine. A failed object without active residue blocks batch completion; failure cannot disappear into a log file.

6. Reconcile batch completion

fa conversion batch complete \
  --file batch-start.json \
  --idempotency-key stored-code-batch-1-complete

Airlift derives the result instead of trusting caller-supplied counts:

• every scoped object needs a terminal attempt;
• every converted attempt needs a matching target artifact digest;
• every failed attempt needs an active residue case;
• every hazardous converted attempt needs its exact-origin conversion_hazard case;
• a mixture of clean and hazardous/failed objects becomes completed_with_residue;
• all failed objects becomes failed.

Hazardous conversions count in the residue population, never the converted population. Completion does not certify parity. Independent validation and readiness policies still decide whether an object can receive a migration certificate.

Conversion hazards: when "converted" is not safe

A deterministic converter can report success while the output is invalid on Databricks or silently lossy. For qualified source profiles, Airlift rescans every converted artifact and records a digest-sealed hazard assessment on the attempt. Assessments carry hazard codes and artifact-relative locations — never SQL text. The Teradata catalog for the exact lakebridge@0.14.2/sqlglot engine covers five codes:

An attempt is unassessed, scan clean, or converted with hazards — and unassessed is never treated as clean. fa conversion list and fa conversion show render the state; the raw assessment travels with --json output.

Findings force a conversion_hazard residue case bound to the exact conversion and assessment digest (its origin). The airlift.conversion_hazard_gate.v1 policy blocks parity certification, migration-certificate minting, and cutover until that exact case is resolved with a repaired target_code artifact and independently reviewed — reviewing an older attempt's case never unlocks a newer attempt. A denied certification request stays denied under its original idempotency key: after the review is approved, issue the certification again with a fresh idempotency key.

Bounded repair agent

The Harness repair agent accepts source SQL, its digest, source dialect, and the deterministic failure. It has no tools, commands, filesystem access, or network access; it produces at most one typed candidate under a fixed source-size, iteration, timeout, retry, and cost bound. Persist that candidate as an artifact, then validate it through Experiments. An agent can create, estimate, and resolve an assigned residue with evidence. It cannot assign commercial work or approve its own resolution.

Before promoting a repair strategy, evaluate a versioned dataset in Experiments and pass the digested summary to evaluateRepairPromotion. Promotion fails closed on sample size, schema validity, compile rate, parity rate, unsafe-output rate, or regression thresholds.

Databricks App workflow

Open Conversion in the Airlift App to create/start/reconcile batches, inspect attempt and artifact lineage, and register artifact references. Open Remediation for failed objects. The App and CLI invoke the same action definitions, policies, idempotency rules, event log, and tenant-scoped projections.