Pipeline and code modernization
Developer workflow for importing orchestration metadata, generating Databricks files where supported, and governing the remaining implementation work.
Pipeline and code modernization
The pipeline compiler routes source constructs to native generation, bounded repair, or human implementation.
A generated candidate is still an artifact that requires deployment and independent validation.
Import a supported descriptor, review construct dispositions, generate the candidate, and register its digest.
A warehouse migration is incomplete when its orchestration stays behind. Airlift treats pipeline graphs, scripts, parameters, schedules, retry behavior, checkpoints, external effects, and downstream consumers as migration objects with the same provenance and evidence discipline as tables and SQL code.
End-to-end flow
| Stage | What you do | What Airlift produces or governs |
|---|---|---|
| Export | Export source metadata without credentials | Native source input or a normalized, content-digested manifest |
| Import and route | Run fa migration-ir import for ADF, or supply a normalized manifest for another profile | Lossless IR, dependency order, dispositions, provenance, and remediation packs |
| Generate | Run fa migration-ir generate for a native generator, or implement the declared targets | Concrete file bodies for ADF; explicit descriptor-only status for other profiles |
| Verify bytes | Run fa migration-ir validate | Artifact-set digest plus per-file content and byte-length verification |
| Implement | Send bounded cases to Harness; assign human cases; review replacement artifacts | Immutable artifact references and an auditable repair history |
| Validate | Execute representative data and control-flow scenarios in Experiments | Independent verdicts for data, branch, retry, failure, and restart behavior |
| Deploy | Register the requirement in Airlift; deploy the immutable release with Runway | Preview, release digest, promotion, reconciliation, and rollback reference |
| Certify | Evaluate object-specific readiness | Signed certificate that identifies exactly what was tested |
| Modernize | Create a separate native redesign release | Measured Lakeflow, serverless, cost, reliability, and operability improvements |
Source-to-target mappings
ADF and Synapse pipelines
ADF is the first native export path. Airlift reads ARM or Git exports, resolves pipeline, dataset, trigger, and activity dependencies, and emits concrete Databricks Workflow, Python, BDD, configuration, and Asset Bundle files. One-input, one-output Copy activities produce a parameterized table materialization task. Mapping Data Flow, expression, condition, loop, and tumbling-window semantics enter bounded repair because source syntax alone cannot prove equivalent behavior. Web activities and other side effects require an engineer to design and approve the replacement.
SSIS
The current SSIS profile accepts a credential-free normalized manifest and classifies Data Flow Tasks, package execution, containers, SQL tasks, and Script Tasks. It produces target declarations and remediation packs, not DTSX parsing or file bodies. A native DTSX/ISPac importer and concrete generator must pass the same no-silent-drop contract before this profile is described as a compiler.
Informatica PowerCenter
The current PowerCenter profile routes normalized mappings, source qualifiers, workflows, expressions, stored-procedure transformations, and external effects. Repository XML import and concrete Lakeflow/Workflow/PySpark generation remain separate adapter work.
SAS
The current SAS profile routes normalized PROC SQL, DATA step, macro, schedule, X command, and host-side-effect nodes. It does not parse SAS source or claim semantic conversion. Native parsing, macro expansion, generated SQL/PySpark, and analytic/model equivalence remain required before SAS receives a compiler claim.
DataStage, Talend, and ODI
These are normalized-manifest routing profiles. They preserve known and unknown constructs, select deterministic, repairable, or human lanes, and state the intended Databricks target. They do not yet parse native exports or materialize those targets.
dbt and Airflow
These are normalized-manifest routing profiles. Native manifest.json and serialized-DAG
importers, SQL/Python file preservation, and concrete Databricks job generation remain
required. Hooks, Bash operators, and host-specific effects stay explicit human remediation.
Behavior is the acceptance contract
Syntax conversion is not completion. For each migrated unit, exercise:
- representative input and output data;
- null, decimal, timestamp, ordering, and duplicate behavior;
- parameters, variables, branches, and loops;
- schedule and trigger behavior;
- retry limits and failure propagation;
- checkpoint, restart, and duplicate-run behavior;
- external side effects and apply-once constraints; and
- runtime, throughput, and cost thresholds.
Only independent admitted verdicts advance readiness. Compiler success and agent output are implementation evidence, not proof of parity.
Migration release before modernization release
First preserve the source behavior needed for safe cutover. Then create a separate modernization item for native redesign: simplify orchestration, adopt serverless, consolidate pipelines, improve observability, or change the data product contract. This keeps the baseline certificate honest and makes modernization value measurable.
Start with the Migration IR command guide, then use Engineering remediation, Validation, and the Runway integration.
Register migration artifacts
Record immutable migration outputs, provider references, content digests, and producer lineage in Airlift.
Operational application modernization
Turn migration findings into portable Databricks Apps and Lakebase releases with governed state, synthetic BDD, branch rehearsal, and evidence-qualified delivery.