Compile ADF, SSIS, PowerCenter, and SAS estates into governed Databricks implementation, validation, deployment, and remediation work.

ETL modernization

This capability treats pipelines and analytics code as first-class migration objects. ADF and Synapse Pipelines have a native export importer and concrete Databricks file generator. SSIS, Informatica PowerCenter, SAS, IBM DataStage, Talend, Oracle Data Integrator, dbt, and Apache Airflow currently have explicit normalized-manifest routing profiles. Those profiles preserve scope and classify work, but do not yet parse native exports or emit deployable file bodies.

fa source inspect etl_modernization
fa source plan etl_modernization --variant informatica
fa source plan etl_modernization --variant azure_data_factory --json
fa migration-ir import --source adf-synapse --file adf-export/ \
  --estate-name "Commerce ADF" --snapshot-at 2030-01-15T12:00:00Z \
  --output migration-ir.json
fa migration-ir generate --file migration-ir.json --out-dir generated
fa migration-ir validate --file generated/artifact-set.json --root generated

What Airlift adds

Phase	Developer work	Governed outcome
Discover	Export pipelines, mappings, connections, parameters, triggers, lineage, and run history	Accepted inventory and dependency graph; credentials replaced by refs
Map and implement	Map source semantics to Lakeflow Jobs, Declarative Pipelines, or Databricks Workflows	Versioned target artifacts and explicit custom-code residue
Rehearse	Run production-like parameters, volumes, dependencies, retries, and checkpoints	Immutable run and reconciliation evidence
Validate	Prove row/business results, schedules, failure behavior, and SLA thresholds	Independent readiness observations
Cut over	Freeze schedule scope and switch orchestration through a certified effector	Authenticated approvals, checkpoint, verification or rollback
Modernize	Optimize serverless compute, lineage, and pipeline design	Separate release and evidence profile

Connection identities are mapped to governed target references. Airlift never imports source passwords from exported pipeline files. Custom components, proprietary connectors, script tasks, implicit variables, and scheduler coupling remain visible in the residue ledger.

The ADF native path has recurring credential-free import, generation, tamper-denial, and materialization coverage. The other eight routing profiles have deterministic fixture coverage for provenance, dependency ordering, dispositions, and no-silent-drop behavior. Neither test level substitutes for live workspace or representative client evidence.

See Migration IR commands for the manifest, routing, and registration contracts and Pipeline and code modernization for the complete developer workflow.

Complete developer command sequence

Generate this exact recipe from the installed CLI so the guide and executable surface stay in sync:

fa source recipe etl_modernization
fa source recipe etl_modernization --variant informatica --json > .airlift/etl_modernization-recipe.json

The App is engagement-aware. ETL and orchestration platforms appears under Active sources only after the source estate is added to an active engagement. The menu is derived from governed engagement scope; installing Airlift does not expose unrelated source pages.

Every remote mutation below requires --host, --org, authenticated workspace identity, and a stable --idempotency-key. JSON request files contain identifiers, artifact references, and opaque credential references—never passwords, tokens, or connection strings. Run fa <resource> <operation> --help for the current schema and exit semantics.

0. Inspect the source contract

fa source inspect etl_modernization --json > .airlift/etl_modernization-profile.json
fa source plan etl_modernization --variant informatica --json > .airlift/etl_modernization-capability-plan.json

Expected artifacts:

.airlift/etl_modernization-profile.json
.airlift/etl_modernization-capability-plan.json

Open Engagements → active engagement in the App. This stage is visible at /engagements after replacing the placeholder ID with the governed engagement ID.

1. Create governed scope and connection references

fa engagement create --file engagement.json --idempotency-key migration-create-v1
fa estate register --file etl_modernization-estate.json --idempotency-key etl_modernization-estate-v1
fa connection register --file etl_modernization-connection.json --idempotency-key etl_modernization-connection-v1
fa engagement update --file etl_modernization-scope.json --idempotency-key etl_modernization-scope-v1
fa engagement preflight <engagement-id>

Expected artifacts:

Governed engagement
Source estate
Opaque connection binding

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id> after replacing the placeholder ID with the governed engagement ID.

2. Assess and accept inventory

fa assessment start --file etl_modernization-assessment-start.json --idempotency-key etl_modernization-assessment-start-v1
fa assessment status <assessment-id> --json
fa assessment record --file etl_modernization-assessment-record.json --idempotency-key etl_modernization-assessment-record-v1
fa assessment accept --file etl_modernization-assessment-accept.json --idempotency-key etl_modernization-assessment-accept-v1
fa inventory list --estate-id <estate-id> --json

Expected artifacts:

Assessment report reference
Normalized inventory
Dependency graph

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/sources/etl_modernization after replacing the placeholder ID with the governed engagement ID.

3. Compile the normalized informatica pipeline manifest

fa migration-ir inspect --file informatica-migration-ir.json
fa migration-ir compile --file informatica-migration-ir.json --json > generated/informatica-routing-plan.json

Expected artifacts:

Validated normalized migration IR
Deterministic routing plan
Source-preserved residue

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/artifacts after replacing the placeholder ID with the governed engagement ID.

Only ADF/Synapse currently has a native export importer and executable file generator. Other ETL variants require an adapter-produced Migration IR manifest and remain descriptor-only.

4. Convert, move, and remediate

fa plan generate --file etl_modernization-migration-plan.json --idempotency-key etl_modernization-plan-v1
fa conversion batch create --file etl_modernization-batch.json --idempotency-key etl_modernization-batch-v1
fa conversion batch start --file conversion-batch-start.json --idempotency-key conversion-start-v1
fa residue list --engagement-id <engagement-id>
fa transfer plan --file etl_modernization-transfer.json --idempotency-key etl_modernization-transfer-v1
fa transfer run <transfer-id> --idempotency-key transfer-run-v1
fa transfer reconcile <transfer-id> --idempotency-key transfer-reconcile-v1

Expected artifacts:

Target artifacts
Residue cases
Transfer checkpoints
Reconciliation evidence

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

5. Validate independently and inspect discrepancies

fa validation run --file etl_modernization-validation.json --idempotency-key etl_modernization-validation-v1
fa validation status <validation-execution-id> --json
fa discrepancy list --engagement-id <engagement-id>
fa artifact list --engagement-id <engagement-id>

Expected artifacts:

Provider run references
Readiness evidence
Discrepancies

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

6. Certify, cut over, and export evidence

fa certificate list --object-id <object-id>
fa cutover status <wave-id> --json
fa evidence list --engagement-id <engagement-id>
fa evidence export --file etl_modernization-evidence-export.json --idempotency-key etl_modernization-evidence-export-v1

Expected artifacts:

Migration certificates
Cutover evidence
Content-digested evidence export

Open Engagements → active engagement in the App. This stage is visible at /assurance after replacing the placeholder ID with the governed engagement ID.

Runway executes releases; Experiments owns validation verdicts; Airlift owns migration readiness and cutover policy.

What developers see in the App

The contextual source workspace shows the accepted estate and the factory stages for this engagement. Artifacts displays immutable references, content digests, media types, and provider lineage. Runs displays assessment, conversion, transfer, validation, and deployment executions without treating a provider's success as an Airlift verdict.

Airlift ETL and orchestration platforms source workspace showing the governed estate and developer command entry point — ETL and orchestration platforms has its own engagement-scoped workspace; unrelated source systems are not shown.
What you are seeing
This public synthetic capture demonstrates Airlift setup and navigation for ETL and orchestration platforms; it is not evidence of a live connection or certified migration.
What to do next
Register the client estate, bind a credential reference, run the ETL and orchestration platforms assessment recipe, and admit the resulting evidence.
Read the developer workflow

These are automated captures from public synthetic engagements. The source workspace is specific to ETL and orchestration platforms; no unrelated source is presented as its migration journey. For sources without an evidence-backed journey, the image demonstrates setup, navigation, and developer entry points only—not a live connection, converted output, or certified migration. No client data, credentials, workspace hostnames, or internal deployment identifiers are embedded in the images.

ETL modernization

On this page