BigQuery to Databricks
Assess and govern BigQuery datasets, routines, scheduled queries, data movement, validation, and cutover.
BigQuery to Databricks
The BigQuery pack inventories datasets, tables, nested schemas, views, routines, scheduled queries, BigQuery ML, remote functions, authorized views, and downstream BI. It combines specialist conversion and transfer tools with Airlift's governed ledger and independent evidence model.
fa source inspect bigquery
fa source plan bigquery
fa source plan bigquery --json > .airlift/bigquery-plan.jsonMigration route
| Stage | Implementation | Airlift value |
|---|---|---|
| Assess | Export metadata and source artifacts; run the admitted analyzer route | Accepted inventory, dependencies, exclusions, immutable report refs |
| Convert | Translate supported GoogleSQL and disposition scripting, ML, remote functions, and authorized-view behavior | Versioned attempts, artifact digests, warnings, residue ownership |
| Transfer | Use restartable storage exports or an admitted connector with timestamp/change catch-up | Manifests, watermarks, restart, lag, reconciliation |
| Validate | Test nested/repeated fields, numeric and timestamp semantics, partitions, access, and business queries | Independent readiness evidence |
| Cut over | Freeze the wave and change consumers with checkpoint/apply-once/verify | Approvals and verifiable outcome or rollback |
| Modernize | Adopt Delta, liquid clustering, Lakeflow, Unity Catalog, and serverless SQL | Separate, independently validated release |
The pack is assessable: it describes an admitted assessment route, but that does not
certify your transfer, validation, scale, or cutover. Produce a live manifest and run:
fa source certification-check bigquery-certification.jsonComplete developer command sequence
Generate this exact recipe from the installed CLI so the guide and executable surface stay in sync:
fa source recipe bigquery
fa source recipe bigquery --variant bigquery --json > .airlift/bigquery-recipe.jsonThe App is engagement-aware. Google BigQuery appears under Active sources only after the source estate is added to an active engagement. The menu is derived from governed engagement scope; installing Airlift does not expose unrelated source pages.
Every remote mutation below requires --host, --org, authenticated workspace
identity, and a stable --idempotency-key. JSON request files contain identifiers,
artifact references, and opaque credential references—never passwords, tokens, or
connection strings. Run fa <resource> <operation> --help for the current schema and
exit semantics.
0. Inspect the source contract
fa source inspect bigquery --json > .airlift/bigquery-profile.json
fa source plan bigquery --variant bigquery --json > .airlift/bigquery-capability-plan.jsonExpected artifacts:
- .airlift/bigquery-profile.json
- .airlift/bigquery-capability-plan.json
Open Engagements → active engagement in the App. This stage is visible at
/engagements after replacing the placeholder ID with the governed engagement ID.
1. Create governed scope and connection references
fa engagement create --file engagement.json --idempotency-key migration-create-v1
fa estate register --file bigquery-estate.json --idempotency-key bigquery-estate-v1
fa connection register --file bigquery-connection.json --idempotency-key bigquery-connection-v1
fa engagement update --file bigquery-scope.json --idempotency-key bigquery-scope-v1
fa engagement preflight <engagement-id>Expected artifacts:
- Governed engagement
- Source estate
- Opaque connection binding
Open Engagements → active engagement in the App. This stage is visible at
/engagements/<engagement-id> after replacing the placeholder ID with the governed engagement ID.
2. Assess and accept inventory
fa assessment start --file bigquery-assessment-start.json --idempotency-key bigquery-assessment-start-v1
fa assessment status <assessment-id> --json
fa assessment record --file bigquery-assessment-record.json --idempotency-key bigquery-assessment-record-v1
fa assessment accept --file bigquery-assessment-accept.json --idempotency-key bigquery-assessment-accept-v1
fa inventory list --estate-id <estate-id> --jsonExpected artifacts:
- Assessment report reference
- Normalized inventory
- Dependency graph
Open Engagements → active engagement in the App. This stage is visible at
/engagements/<engagement-id>/sources/bigquery after replacing the placeholder ID with the governed engagement ID.
3. Implement the admitted source adapter
No source-specific executable compiler exists in this release. Continue with assessment, governed work, and an admitted adapter; Airlift does not invent executable output.
Expected artifacts:
- Capability plan and adapter requirements only
Open Engagements → active engagement in the App. This stage is visible at
/engagements/<engagement-id>/artifacts after replacing the placeholder ID with the governed engagement ID.
This source is cataloged or assessable but has no source-specific executable compiler in the installed Airlift generation.
4. Convert, move, and remediate
fa plan generate --file bigquery-migration-plan.json --idempotency-key bigquery-plan-v1
fa conversion batch create --file bigquery-batch.json --idempotency-key bigquery-batch-v1
fa conversion batch start --file conversion-batch-start.json --idempotency-key conversion-start-v1
fa residue list --engagement-id <engagement-id>
fa transfer plan --file bigquery-transfer.json --idempotency-key bigquery-transfer-v1
fa transfer run <transfer-id> --idempotency-key transfer-run-v1
fa transfer reconcile <transfer-id> --idempotency-key transfer-reconcile-v1Expected artifacts:
- Target artifacts
- Residue cases
- Transfer checkpoints
- Reconciliation evidence
Open Engagements → active engagement in the App. This stage is visible at
/engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.
5. Validate independently and inspect discrepancies
fa validation run --file bigquery-validation.json --idempotency-key bigquery-validation-v1
fa validation status <validation-execution-id> --json
fa discrepancy list --engagement-id <engagement-id>
fa artifact list --engagement-id <engagement-id>Expected artifacts:
- Provider run references
- Readiness evidence
- Discrepancies
Open Engagements → active engagement in the App. This stage is visible at
/engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.
6. Certify, cut over, and export evidence
fa certificate list --object-id <object-id>
fa cutover status <wave-id> --json
fa evidence list --engagement-id <engagement-id>
fa evidence export --file bigquery-evidence-export.json --idempotency-key bigquery-evidence-export-v1Expected artifacts:
- Migration certificates
- Cutover evidence
- Content-digested evidence export
Open Engagements → active engagement in the App. This stage is visible at
/assurance after replacing the placeholder ID with the governed engagement ID.
Runway executes releases; Experiments owns validation verdicts; Airlift owns migration readiness and cutover policy.
What developers see in the App
The contextual source workspace shows the accepted estate and the factory stages for this engagement. Artifacts displays immutable references, content digests, media types, and provider lineage. Runs displays assessment, conversion, transfer, validation, and deployment executions without treating a provider's success as an Airlift verdict.
Google BigQuery has its own engagement-scoped workspace; unrelated source systems are not shown.
This public synthetic capture demonstrates Airlift setup and navigation for Google BigQuery; it is not evidence of a live connection or certified migration.
Register the client estate, bind a credential reference, run the Google BigQuery assessment recipe, and admit the resulting evidence.
These are automated captures from public synthetic engagements. The source workspace is specific to Google BigQuery; no unrelated source is presented as its migration journey. For sources without an evidence-backed journey, the image demonstrates setup, navigation, and developer entry points only—not a live connection, converted output, or certified migration. No client data, credentials, workspace hostnames, or internal deployment identifiers are embedded in the images.