Pipeline import and generation

fa migration-ir gives migration engineers a reproducible path from source orchestration metadata to reviewable Databricks implementation files. It does not treat a list of target paths as generated code and it does not call a routing policy a native compiler.

The command group has two capability levels:

The eight normalized-manifest profiles are useful for inventory, dependency ordering, effort routing, and human remediation. Their artifact declarations are not file bodies. Only ADF/Synapse currently has the native importer and concrete file generator described below.

ADF end-to-end workflow

fa migration-ir import \
  --source adf-synapse \
  --file factory-export/ \
  --estate-name "Commerce integration" \
  --snapshot-at 2030-01-15T12:00:00Z \
  --output migration-ir.json

fa migration-ir inspect --file migration-ir.json

fa migration-ir generate \
  --file migration-ir.json \
  --out-dir generated

fa migration-ir validate \
  --file generated/artifact-set.json \
  --root generated

The generated directory contains the actual bytes referenced by the artifact set:

generated/
├── artifact-set.json
├── databricks.yml
├── resources/
│   └── adf-migration.yml
├── src/adf/
│   ├── copy_runtime.py
│   └── review_required.py
├── generated/adf_synapse/
│   └── ... content-digested node files
└── tests/
    └── ... Experiments-compatible BDD specifications

artifact-set.json includes every file body, SHA-256 digest, byte length, source-node provenance, required runtime binding, limitation, and executable flag. You can transport that one JSON document through CI and materialize it later:

fa migration-ir materialize \
  --file artifact-set.json \
  --out-dir generated

Import native ADF exports

Use an ARM template, a single ADF Git JSON file, or the root of an ADF Git export:

fa migration-ir import \
  --source adf-synapse \
  --file adf-export/ \
  --estate-name "Finance pipelines" \
  --snapshot-at 2030-01-15T12:00:00Z \
  --product-version 2018-06-01 \
  --output migration-ir.json

The importer reads pipeline/, dataset/, linkedService/, and trigger/ documents. It creates stable node identifiers, resolves activity and dataset dependencies, preserves source fragments and locations, and creates opaque connection bindings.

Provide explicit connection references when the generated defaults do not match your connection registry:

{
  "WarehouseSource": "databricks-connection://migration/warehouse-source",
  "LandingStorage": "databricks-secret://migration/landing-storage"
}

fa migration-ir import \
  --source adf-synapse \
  --file adf-export/ \
  --estate-name "Finance pipelines" \
  --snapshot-at 2030-01-15T12:00:00Z \
  --credential-map credential-refs.json \
  --output migration-ir.json

Never add credentials to the export. Parameter, Key Vault, and secure-string references become opaque bindings. A plaintext password, token, connection string, account key, or client secret fails import before an IR is produced.

Inspect and compile routing

fa migration-ir inspect --file migration-ir.json
fa migration-ir compile --file migration-ir.json --json > migration-ir-bundle.json

Inspection validates IDs, dependencies, and cycles, then reports routing counts. Generic compilation produces:

• one typed IR node per accepted source construct;
• deterministic, agent_repairable, human_only, excluded, or blocked disposition;
• source fragments, locations, extractor generation, and content digests;
• artifact declarations with materialization: descriptor_only;
• behavioral-test requirements and remediation packs; and
• ownership contracts for Harness, Experiments, Platform, and Runway.

Unknown nodes are not dropped. Airlift retains their safe source fragment and routes them to remediation. A new rule can therefore be developed against the exact accepted source construct without rescanning the estate.

Generate concrete ADF files

fa migration-ir generate \
  --file migration-ir.json \
  --out-dir generated \
  --artifact-set generated/artifact-set.json

Generation currently admits the common ADF patterns it can express safely:

• pipeline and dataset configuration;
• one-input, one-output Copy activities;
• pipeline dependencies and retry counts;
• supported schedule triggers;
• explicit target-job bindings for pipeline invocation;
• a Databricks Asset Bundle root and Workflow resource; and
• one BDD specification per deterministic node.

ADF expressions, Mapping Data Flow semantics, conditions, loops, tumbling-window behavior, external calls, and unknown activities do not disappear. They remain repairable or human work until an implementation artifact is independently reviewed and validated.

The generated copy runtime requires Unity Catalog source_identifier and target_identifier bindings. It covers table-to-table materialization. Snapshot, CDC, delete, file-layout, and schema-drift behavior belongs to the selected Airlift transfer profile and must be validated separately.

Validate bytes

Validate the artifact-set envelope by itself:

fa migration-ir validate --file generated/artifact-set.json

Also compare every materialized file with its recorded body, digest, and byte length:

fa migration-ir validate \
  --file generated/artifact-set.json \
  --root generated

Validation fails on a modified, missing, duplicated, or path-escaping artifact. It does not claim that Databricks runtime behavior is correct. Use Runway preview and deployment, then execute the generated BDD and parity scenarios through Fabric Experiments.

Register immutable lineage

After validation, store the artifact set in an admitted immutable store and register its reference:

fa migration-ir register \
  --file generated/artifact-set.json \
  --engagement-id <engagement-id> \
  --estate-id <estate-id> \
  --artifact-id <immutable-artifact-reference> \
  --idempotency-key <stable-key>

Registration invokes airlift.artifact_register through the authenticated Platform action API. Airlift records the engagement, estate, reference, media type, tool generation, and artifact-set digest. It does not put file bodies or credentials in the migration ledger.

Qualify a release candidate

qualify turns byte verification, runtime bindings, residue decisions, Databricks bundle validation, and optional live provider evidence into one fail-closed report:

fa migration-ir qualify \
  --file generated/artifact-set.json \
  --root generated \
  --bindings release-bindings.json \
  --resolutions residue-resolutions.json \
  --proof hermetic_proven \
  --output qualification.json

For workspace proof, add --workspace-evidence workspace-evidence.json and request --proof workspace_proven. The evidence must contain the governed Airlift artifact, deployment requirement, validation execution, per-object validation runs, and matching Databricks workspace references. A missing or mismatched Runway/Experiments reference, artifact digest, tenant, object verdict, or workspace identity blocks the command. Validation runs exported with a conversionId require "schemaVersion": 2 on the document; untagged documents are read as version 1 and keep working unchanged.

Read Qualify a generated release for the binding, residue, Runway, Experiments, and workspace-evidence contracts.

Deploy and validate

The generated bundle is a candidate release:

1. Use fa migration-ir validate and fa migration-ir qualify to prove byte integrity, bundle validation, runtime bindings, and residue disposition.
2. Use Airlift conversion and remediation actions to resolve every non-deterministic node.
3. Use fr to preview, deploy, promote, reconcile, or roll back the immutable release.
4. Run generated behavioral scenarios with Fabric Experiments.
5. Return Runway and Experiments references and digests to Airlift readiness.

fa owns migration intent, artifact lineage, readiness, and cutover gates. fr owns release execution. There is no fa migration-ir deploy command.