Build a Synapse-to-Databricks migration

Airlift’s Synapse path combines a governed warehouse plan with a native ADF export importer and concrete Databricks file generator. The outputs are not a presentation checklist: they include dependency order, target mappings, materialized Workflow, Python, BDD, and Asset Bundle files, transfer proof requirements, explicit repair work, and immutable digests used by the migration ledger.

Use it for dedicated SQL pools, serverless SQL pools, or mixed Synapse workspaces. The Lakebridge handles admitted SQL analysis and conversion. Airlift inventories data, external storage, workloads, permissions, pipelines, linked services, triggers, notebooks, and downstream consumers, then controls how implementation and evidence converge.

What Airlift adds to a Synapse migration

Airlift owns the migration ledger and the decisions that advance it. It does not copy a SQL transpiler, deployment engine, or test engine into the application. That boundary is what lets a developer replace an adapter without losing governance or historical evidence.

1. Prepare the manifest

Create synapse-manifest.json. Do not put credentials, tokens, or connection strings in this file.

{
  "schemaVersion": 1,
  "estate": {
    "name": "Finance warehouse",
    "variant": "synapse_dedicated_sql",
    "synapseVersion": "10.0",
    "sourceSnapshot": "2030-01-15T12:00:00.000Z"
  },
  "inventory": [
    {
      "sourceId": "schema:finance",
      "name": "finance",
      "kind": "schema",
      "sourcePath": "finance",
      "owner": "finance-data"
    },
    {
      "sourceId": "table:finance/ledger",
      "name": "ledger",
      "kind": "table",
      "sourcePath": "finance.ledger",
      "owner": "finance-data",
      "dependencies": ["schema:finance"],
      "distribution": "hash",
      "distributionColumns": ["account_id"],
      "partitionColumns": ["posting_date"]
    }
  ],
  "adfExport": { "resources": [] }
}

Supported inventory kinds are schema, table, view, stored_procedure, function, external_table, copy_statement, polybase_object, workload_group, permission, pipeline, linked_service, trigger, notebook, and consumer.

Mark retired objects with dead: true. Use excludedReason for an explicit scope decision. Add unsupportedReasons to route known dynamic SQL, cross-database behavior, side effects, or ambiguous distribution logic into remediation.

2. Inspect before creating work

fa source inspect synapse
fa synapse inspect --file synapse-manifest.json

The command validates references, rejects duplicate IDs and missing dependencies, and reports in-scope inventory, ETL task count, residue count, human-residue count, and the prospective bundle digest. A dependency cycle fails the command instead of silently inventing an execution order.

To preview task routing without materializing files:

fa synapse compile-adf --file TemplateForWorkspace.json --json

That command returns execution-neutral task mappings only. To import a native export and generate actual file bodies, use the migration IR workflow:

fa migration-ir import --source adf-synapse \
  --file TemplateForWorkspace.json \
  --estate-name "Finance pipelines" \
  --snapshot-at 2030-01-15T12:00:00Z \
  --output migration-ir.json
fa migration-ir generate --file migration-ir.json --out-dir generated
fa migration-ir validate --file generated/artifact-set.json --root generated

Linked services become credential-binding requirements; credential properties are never copied into IR. Unknown activities remain source-preserved remediation rather than being silently dropped.

3. Generate and register the bundle

fa synapse plan --file synapse-manifest.json --json > synapse-plan.json

Store synapse-plan.json in your admitted immutable artifact store. Then register its reference with the active engagement and Synapse estate:

fa synapse register \
  --file synapse-plan.json \
  --engagement-id eng_01ARZ3NDEKTSV4RRFFQ69G5FAV \
  --estate-id est_01ARZ3NDEKTSV4RRFFQ69G5FAV \
  --artifact-id volumes/migration/plans/synapse-plan.json \
  --idempotency-key finance-synapse-plan-v1

register invokes airlift.artifact_register; it does not upload the file. The API derives organization and actor identity from the authenticated principal. The artifact reference and the bundle share the same digest, so replacing the referenced content is detectable.

4. Execute the factory

Before accepting generated workbench artifacts, inspect the exact repository route:

fa source constructs synapse \
  --variant synapse_dedicated_sql \
  --construct adf.copy \
  --artifact-kind target_code

The construct catalog is the shared source for Synapse/ADF assessment lanes, Migration IR routing, and v2 artifact admission. A new artifact whose declared construct, disposition, or target disagrees with the exact catalog row is rejected without an event. Historical rows remain replayable and visible, but the App marks their route as diverged or unrecognized so an operator can regenerate or review them. Catalog alignment is implementation metadata; it does not create a capability claim or prove workspace behavior.

Open Active sources → Azure Synapse inside the active engagement, or use the corresponding CLI surfaces:

fa assessment list --estate-id <estate>
fa plan list --engagement-id <engagement>
fa conversion batch list --engagement-id <engagement>
fa residue list --engagement-id <engagement>
fa transfer list --estate-id <estate>
fa deployment list --estate-id <estate>
fa validation list --estate-id <estate>
fa cutover list --estate-id <estate>
fa modernization list --engagement-id <engagement>
fa engagement status <engagement>
fa engagement status <engagement> --json

The engagement status view shows Azure Synapse → Databricks Lakehouse, all eight delivery phases, the number of governed objects at each stage, open human residue, validation blockers, active certificates, and the next action. It derives these values from admitted records; developers do not edit the percentage.

The generated bundle tells these stages what must exist; the governed actions record what actually happened. Converter success never certifies an object. Deployment success never proves parity. A certificate is minted only after the current validation profile has admitted evidence for every required track.

Target and movement behavior

Airlift proposes Unity Catalog schemas and grants, Delta tables, external locations, Lakeflow pipelines and jobs, Databricks SQL assets, notebooks, connection bindings, validation assets, and Databricks Asset Bundle resources. Runway owns deployment, promotion, and rollback and returns immutable release references to Airlift.

The Synapse transfer profile is snapshot plus incremental catch-up. Before completion, the project must record stable snapshot identity, source watermarks, deletes, late data, schema drift, restart checkpoints, bounded concurrency, throughput, rejects, and reconciliation. These are executable evidence requirements, not prose reminders.

Human remediation is part of the product

Cross-database transactions, external side effects, custom activities, and ambiguous business semantics are intentionally visible. Airlift creates priced, assignable residue instead of hiding that work inside an automation percentage. A specialist can repair the artifact, record independent validation, and submit it for human review without bypassing the certificate policy.

Security properties

• source exports contain metadata and code, never live credentials;
• linked services are represented as opaque connection-binding requirements;
• every remote mutation uses authenticated identity, tenant scope, policy, audit, and a stable idempotency key;
• Airlift references Experiments and Runway results by ID and digest and never re-owns their state;
• cutover remains blocked until frozen scope, approvals, fresh evidence, and effector certification all pass.

See Synapse CLI commands for command and exit semantics.

Read a development journey

When a team exercises the Synapse factory before production reviewers and effectors are available, Airlift records the result as Development evidence. Use the CLI and App to inspect what actually passed:

fa engagement status <engagement-id>
fa validation list --engagement-id <engagement-id>
fa validation runs --object-id <object-id>
fa certificate list --object-id <object-id>
fa transfer list --engagement-id <engagement-id>
fa cutover status <wave-id>

In the App, open the engagement’s Migration status, Artifacts, and Run ledger; then open Assurance center and Cutover control. A complete development exercise should show converted and development-certified objects, a completed validation execution, a reconciled transfer, and a rehearsed wave. It should also show that production is still blocked. See Development assurance for the exact boundary and expected UI state.

Prove generation without a source account

Run import, generate, and validate against a credential-free ARM or Git export. This proves parsing, dependency preservation, file materialization, and digest integrity. It is account-independent, so the maximum result is hermetic_proven; it does not prove source connectivity or Databricks behavior.

Add live workspace execution, Experiments verdicts, a Runway release, transfer restart evidence, and current provider generations before promoting the relevant capability cells to workspace_proven. Representative client behavior and production cutover remain separate evidence levels. See Capability certification for the proof model.

Run the authenticated sandbox journey

The isolated sandbox can execute the same account-independent provider through the real Temporal parent workflow and show durable progress in the engagement Run ledger. An administrator must first prepare one exact governed candidate in the shared PostgreSQL ledger:

• one active engagement bound to one Synapse estate;
• one verified Synapse source connection;
• one planned transfer whose non-empty object scope belongs to that estate;
• one requested deployment whose artifacts cover exactly the transfer objects;
• assigned validation profiles for those objects.

Open Engagements → engagement → Run ledger and select Start Synapse journey. The server accepts only organization and engagement from the form, resolves the provider IDs from canonical projections, verifies the signed-in natural person and governed transfer permission, and starts or attaches to a request-bound workflow on airlift-sandbox-v1. An exact retry of the same browser request collapses to the open workflow; a later retry after terminal failure receives a fresh workflow and run-ledger row for the same governed scope. Temporal activities record discovering, assessing, converting, transferring, deploying, validation, and terminal state through Platform Host. Refreshing or reconnecting reads the same replayable PostgreSQL projection.

After deployment the journey pauses at awaiting_validation_request. This is deliberate: the worker does not manufacture operator authority. Create the exact governed validation request in Validation, then wake the existing workflow through the supported validation control. The hermetic provider cannot mint business acceptance, production certification, or a client cutover decision.

The browser lane has the same permanent maximum result as the CLI lane: hermetic_proven. Seeing a completed workflow in the sandbox is not live Synapse or Databricks workspace evidence.

Certify and cut over

Use object-type-specific profiles for tables, SQL routines, pipelines, permissions, and consumers. Airlift mints signed certificates only from current admitted validation evidence. Cutover then rechecks the frozen wave, certificates, approvals, deployment digests, operational window, and certified effector before it permits an apply-once effect. Failure and uncertainty retain a governed rollback or reconciliation path.

Complete developer command sequence

Generate this exact recipe from the installed CLI so the guide and executable surface stay in sync:

fa source recipe synapse
fa source recipe synapse --variant synapse_dedicated_sql --json > .airlift/synapse-recipe.json

The App is engagement-aware. Azure Synapse appears under Active sources only after the source estate is added to an active engagement. The menu is derived from governed engagement scope; installing Airlift does not expose unrelated source pages.

Every remote mutation below requires --host, --org, authenticated workspace identity, and a stable --idempotency-key. JSON request files contain identifiers, artifact references, and opaque credential references—never passwords, tokens, or connection strings. Run fa <resource> <operation> --help for the current schema and exit semantics.

0. Inspect the source contract

fa source inspect synapse --json > .airlift/synapse-profile.json
fa source plan synapse --variant synapse_dedicated_sql --json > .airlift/synapse-capability-plan.json

Expected artifacts:

• .airlift/synapse-profile.json
• .airlift/synapse-capability-plan.json

Open Engagements → active engagement in the App. This stage is visible at /engagements after replacing the placeholder ID with the governed engagement ID.

1. Create governed scope and connection references

fa engagement create --file engagement.json --idempotency-key migration-create-v1
fa estate register --file synapse-estate.json --idempotency-key synapse-estate-v1
fa connection register --file synapse-connection.json --idempotency-key synapse-connection-v1
fa engagement update --file synapse-scope.json --idempotency-key synapse-scope-v1
fa engagement preflight <engagement-id>

Expected artifacts:

• Governed engagement
• Source estate
• Opaque connection binding

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id> after replacing the placeholder ID with the governed engagement ID.

2. Assess and accept inventory

fa assessment start --file synapse-assessment-start.json --idempotency-key synapse-assessment-start-v1
fa assessment status <assessment-id> --json
fa assessment record --file synapse-assessment-record.json --idempotency-key synapse-assessment-record-v1
fa assessment accept --file synapse-assessment-accept.json --idempotency-key synapse-assessment-accept-v1
fa inventory list --estate-id <estate-id> --json

Expected artifacts:

• Assessment report reference
• Normalized inventory
• Dependency graph

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/sources/synapse after replacing the placeholder ID with the governed engagement ID.

3. Compile Synapse and ADF artifacts

fa synapse inspect --file synapse-manifest.json
fa synapse plan --file synapse-manifest.json --json > generated/synapse-plan.json
fa migration-ir import --source adf-synapse --file TemplateForWorkspace.json --estate-name "Source pipelines" --snapshot-at <iso-timestamp> --output generated/migration-ir.json
fa migration-ir generate --file generated/migration-ir.json --out-dir generated
fa migration-ir validate --file generated/artifact-set.json --root generated

Expected artifacts:

• Synapse plan
• Databricks Asset Bundle
• Workflow/Python files
• BDD specifications
• Artifact set digest

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/artifacts after replacing the placeholder ID with the governed engagement ID.

4. Convert, move, and remediate

fa plan generate --file synapse-migration-plan.json --idempotency-key synapse-plan-v1
fa conversion batch create --file synapse-batch.json --idempotency-key synapse-batch-v1
fa conversion batch start --file conversion-batch-start.json --idempotency-key conversion-start-v1
fa residue list --engagement-id <engagement-id>
fa transfer plan --file synapse-transfer.json --idempotency-key synapse-transfer-v1
fa transfer run <transfer-id> --idempotency-key transfer-run-v1
fa transfer reconcile <transfer-id> --idempotency-key transfer-reconcile-v1

Expected artifacts:

• Target artifacts
• Residue cases
• Transfer checkpoints
• Reconciliation evidence

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

5. Validate independently and inspect discrepancies

fa validation run --file synapse-validation.json --idempotency-key synapse-validation-v1
fa validation status <validation-execution-id> --json
fa discrepancy list --engagement-id <engagement-id>
fa artifact list --engagement-id <engagement-id>

Expected artifacts:

• Provider run references
• Readiness evidence
• Discrepancies

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

6. Certify, cut over, and export evidence

fa certificate list --object-id <object-id>
fa cutover status <wave-id> --json
fa evidence list --engagement-id <engagement-id>
fa evidence export --file synapse-evidence-export.json --idempotency-key synapse-evidence-export-v1

Expected artifacts:

• Migration certificates
• Cutover evidence
• Content-digested evidence export

Open Engagements → active engagement in the App. This stage is visible at /assurance after replacing the placeholder ID with the governed engagement ID.

Runway executes releases; Experiments owns validation verdicts; Airlift owns migration readiness and cutover policy.

What developers see in the App

The contextual source workspace shows the accepted estate and the factory stages for this engagement. Artifacts displays immutable references, content digests, media types, and provider lineage. Runs displays assessment, conversion, transfer, validation, and deployment executions without treating a provider's success as an Airlift verdict.

View full size

View full size

View full size

View full size

These are automated captures from public synthetic engagements. The source workspace is specific to Azure Synapse; no unrelated source is presented as its migration journey. For sources without an evidence-backed journey, the image demonstrates setup, navigation, and developer entry points only—not a live connection, converted output, or certified migration. No client data, credentials, workspace hostnames, or internal deployment identifiers are embedded in the images.