Implement Hadoop migrations by decomposing Hive, Impala, Oozie, Pig, Sqoop, Spark, MapReduce, storage, metastore, and security workloads.

Migrate Hadoop with Airlift

Hadoop is not one SQL dialect. Airlift models it as an estate containing Hive and Impala SQL, Oozie, Pig, Sqoop, Spark and MapReduce jobs, HDFS or object storage, Hive Metastore, Ranger/Sentry policy, schedules, and downstream consumers. Each construct gets its own conversion and validation disposition.

What Airlift adds to a Hadoop migration

Stage	Databricks tool or project adapter	Airlift responsibility
discover	Lakebridge Analyzer runs per Hive, Impala, Oozie, Pig, and Sqoop surface plus filesystem/metastore exports	accept one connected inventory, dependencies, exclusions, source versions, and report digests
plan	workload disposition and storage analysis inform target architecture	assign owners and dependency-aware waves; preserve copy, register, rewrite, retire, and human-work decisions
convert	deterministic SQL where supported plus construct-specific rewrites	record source and target artifacts, tool/model/human provenance, warnings, and residue lanes without calling redesign parity
move data	Unity Catalog registration, copy, Delta conversion, or source-specific incremental drivers	track manifests, watermarks, partition and format behavior, lag, restart checkpoints, counts, and reconciliation
validate	admitted Experiments scenarios for tables, workflows, jobs, formats, security, and scale	admit independent workload-specific evidence because no universal Hadoop reconcile connector exists
certify	Airlift evaluates each object's active readiness profile	mint signed certificates identifying artifacts, snapshots, evidence, and policy per workload class
cut over	project metastore, scheduler, policy, job, data-path, and consumer effectors	freeze scope, enforce approvals, checkpoint, apply once, verify, and retain rollback evidence
modernize	Unity Catalog, Delta, Lakeflow, supported runtimes, and serverless work	keep architecture redesign in a separately validated release

This is why Airlift treats Hadoop as a workload portfolio rather than a dialect: data, metastore, policy, orchestration, libraries, jobs, and consumers can require different routes but must still cut over as one dependency-aware estate.

Compile the executable migration pack

Export Hive and Impala definitions, metastore and filesystem inventories, Oozie, Pig and Sqoop projects, Spark and MapReduce jobs, UDF/SerDe metadata, Ranger or Sentry policy, schedules, and consumers. Select cloudera, hortonworks, or apache_hadoop, then run:

fa migration-pack inspect --file hadoop-manifest.json
fa migration-pack plan --file hadoop-manifest.json --json > hadoop-plan.json
fa migration-pack register --file hadoop-plan.json \
  --engagement-id <engagement-id> --estate-id <estate-id> \
  --artifact-id <immutable-artifact-reference> --idempotency-key <stable-key>

The compiler detects incomplete dependencies and cycles, chooses register, convert, bounded repair, or human-remediation routes, and emits a file-manifest/ingestion- watermark transfer contract with workload-specific validation and deployment requirements. See migration-pack commands for the complete manifest schema.

Inspect and generate a plan

fa source inspect hadoop
fa source plan hive
fa source plan hadoop --json > .airlift/hadoop-plan.json

The plan intentionally reports that no single deterministic dialect covers the estate.

Analyze by workload

Export repository and filesystem metadata, then run Analyzer for each surface in scope:

databricks labs lakebridge analyze \
  --source-directory ./source-export/hive \
  --source-tech "Hive" \
  --report-file ./artifacts/hive-analysis.xlsx \
  --generate-json true

databricks labs lakebridge analyze \
  --source-directory ./source-export/oozie \
  --source-tech "Oozie" \
  --report-file ./artifacts/oozie-analysis.xlsx \
  --generate-json true

Repeat for Cloudera (Impala), PIG, and SQOOP. Register SQL, workflows, transfers, custom jobs, storage assets, policies, and consumers as linked migration objects.

Route conversion by construct

Source construct	Target path
Hive or Impala SQL	deterministic SQL conversion where supported, otherwise bounded repair/human lane
Oozie workflows	Lakeflow Jobs or Databricks Workflows design
Pig	DataFrame, SQL, or declarative pipeline rewrite
Sqoop	Lakeflow Connect, Auto Loader, federation, or client transfer adapter
Spark	runtime/library/API upgrade with behavior and scale tests
MapReduce	Spark, SQL, or purpose-built redesign
Custom SerDe/InputFormat/UDF	explicit compatibility implementation and test profile

Do not record a modernization rewrite as deterministic parity. Keep its source, target, prompt/model or human provenance, and validation profile explicit.

Move and validate data

Classify data by location and format. Some objects can be registered through Unity Catalog; others require copy and Delta conversion. Prove partition semantics, file formats, late data, incremental ingestion, restart, duplicate handling, metastore mapping, access-policy denial, workflow ordering, UDF behavior, and representative scale.

Lakebridge Reconcile does not provide a universal Hadoop connector. Use admitted Experiments scenarios and the Airlift transfer evidence contract. Tables cannot pass data readiness from counts alone when partition, format, or late-data behavior matters.

Certify and cut over

Assign validation profiles by workload class rather than certifying the estate from a single table report. Require evidence for metastore mappings, partition and format semantics, workflows, jobs, custom code, policies, performance, and downstream consumers. The project effectors should switch catalogs, schedules, data paths, policies, jobs, and connections through rehearsed waves with explicit rollback or compensation.

Modernize after parity

Move governance to Unity Catalog, data to Delta where appropriate, orchestration to Lakeflow, and workloads to supported Databricks runtimes or serverless compute. Retain the baseline certificate and validate the modernization as a new release.

Complete developer command sequence

Generate this exact recipe from the installed CLI so the guide and executable surface stay in sync:

fa source recipe hadoop
fa source recipe hadoop --variant cloudera --json > .airlift/hadoop-recipe.json

The App is engagement-aware. Hadoop appears under Active sources only after the source estate is added to an active engagement. The menu is derived from governed engagement scope; installing Airlift does not expose unrelated source pages.

Every remote mutation below requires --host, --org, authenticated workspace identity, and a stable --idempotency-key. JSON request files contain identifiers, artifact references, and opaque credential references—never passwords, tokens, or connection strings. Run fa <resource> <operation> --help for the current schema and exit semantics.

0. Inspect the source contract

fa source inspect hadoop --json > .airlift/hadoop-profile.json
fa source plan hadoop --variant cloudera --json > .airlift/hadoop-capability-plan.json

Expected artifacts:

.airlift/hadoop-profile.json
.airlift/hadoop-capability-plan.json

Open Engagements → active engagement in the App. This stage is visible at /engagements after replacing the placeholder ID with the governed engagement ID.

1. Create governed scope and connection references

fa engagement create --file engagement.json --idempotency-key migration-create-v1
fa estate register --file hadoop-estate.json --idempotency-key hadoop-estate-v1
fa connection register --file hadoop-connection.json --idempotency-key hadoop-connection-v1
fa engagement update --file hadoop-scope.json --idempotency-key hadoop-scope-v1
fa engagement preflight <engagement-id>

Expected artifacts:

Governed engagement
Source estate
Opaque connection binding

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id> after replacing the placeholder ID with the governed engagement ID.

2. Assess and accept inventory

fa assessment start --file hadoop-assessment-start.json --idempotency-key hadoop-assessment-start-v1
fa assessment status <assessment-id> --json
fa assessment record --file hadoop-assessment-record.json --idempotency-key hadoop-assessment-record-v1
fa assessment accept --file hadoop-assessment-accept.json --idempotency-key hadoop-assessment-accept-v1
fa inventory list --estate-id <estate-id> --json

Expected artifacts:

Assessment report reference
Normalized inventory
Dependency graph

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/sources/hadoop after replacing the placeholder ID with the governed engagement ID.

3. Compile the hadoop migration pack

fa migration-pack inspect --file hadoop-manifest.json
fa migration-pack plan --file hadoop-manifest.json --json > generated/hadoop-plan.json

Expected artifacts:

Dependency-aware migration pack
Transfer requirements
Validation requirements
Residue lanes

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/artifacts after replacing the placeholder ID with the governed engagement ID.

4. Convert, move, and remediate

fa plan generate --file hadoop-migration-plan.json --idempotency-key hadoop-plan-v1
fa conversion batch create --file hadoop-batch.json --idempotency-key hadoop-batch-v1
fa conversion batch start --file conversion-batch-start.json --idempotency-key conversion-start-v1
fa residue list --engagement-id <engagement-id>
fa transfer plan --file hadoop-transfer.json --idempotency-key hadoop-transfer-v1
fa transfer run <transfer-id> --idempotency-key transfer-run-v1
fa transfer reconcile <transfer-id> --idempotency-key transfer-reconcile-v1

Expected artifacts:

Target artifacts
Residue cases
Transfer checkpoints
Reconciliation evidence

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

5. Validate independently and inspect discrepancies

fa validation run --file hadoop-validation.json --idempotency-key hadoop-validation-v1
fa validation status <validation-execution-id> --json
fa discrepancy list --engagement-id <engagement-id>
fa artifact list --engagement-id <engagement-id>

Expected artifacts:

Provider run references
Readiness evidence
Discrepancies

Open Engagements → active engagement in the App. This stage is visible at /engagements/<engagement-id>/runs after replacing the placeholder ID with the governed engagement ID.

6. Certify, cut over, and export evidence

fa certificate list --object-id <object-id>
fa cutover status <wave-id> --json
fa evidence list --engagement-id <engagement-id>
fa evidence export --file hadoop-evidence-export.json --idempotency-key hadoop-evidence-export-v1

Expected artifacts:

Migration certificates
Cutover evidence
Content-digested evidence export

Open Engagements → active engagement in the App. This stage is visible at /assurance after replacing the placeholder ID with the governed engagement ID.

Runway executes releases; Experiments owns validation verdicts; Airlift owns migration readiness and cutover policy.

What developers see in the App

The contextual source workspace shows the accepted estate and the factory stages for this engagement. Artifacts displays immutable references, content digests, media types, and provider lineage. Runs displays assessment, conversion, transfer, validation, and deployment executions without treating a provider's success as an Airlift verdict.

Airlift Hadoop source workspace showing the governed estate and developer command entry point — Hadoop has its own engagement-scoped workspace; unrelated source systems are not shown.
What you are seeing
This public synthetic capture demonstrates Airlift setup and navigation for Hadoop; it is not evidence of a live connection or certified migration.
What to do next
Register the client estate, bind a credential reference, run the Hadoop assessment recipe, and admit the resulting evidence.
Read the developer workflow

These are automated captures from public synthetic engagements. The source workspace is specific to Hadoop; no unrelated source is presented as its migration journey. For sources without an evidence-backed journey, the image demonstrates setup, navigation, and developer entry points only—not a live connection, converted output, or certified migration. No client data, credentials, workspace hostnames, or internal deployment identifiers are embedded in the images.

Hadoop

On this page