What is the difference between Delta Lake, Apache Iceberg, and Apache Hudi as open table formats?

Delta Lake, Apache Iceberg, and Apache Hudi are all open table formats that add ACID transaction support, schema evolution, and time-travel query capabilities to Parquet files stored in object storage. They differ in how they implement these features, which determines the tradeoffs in schema evolution safety, concurrent write behavior, query latency, and vendor dependency. Delta Lake stores its transaction log as a sequence of JSON files in a _delta_log directory co-located with the data. The Databricks-managed version of Delta Lake adds proprietary extensions (Change Data Feed, liquid clustering, V-ORDER write optimization) that are not part of the open specification and create coupling to the Databricks platform. Apache Iceberg uses a metadata tree (a manifest file that points to data files, plus a snapshot file that points to manifests) stored in the table's metadata directory. Iceberg was designed from the start for multi-engine compatibility: the same Iceberg table can be read by Spark, Trino, Flink, Dremio, Snowflake (via external tables), and BigQuery (via Biglake) without format conversion. Iceberg uses column IDs rather than column names for schema evolution, enabling safe column renames and reorders without breaking readers. Hudi (Hadoop Upsert Delete Incremental) was designed primarily for record-level upsert workloads — ingesting change data capture streams from relational databases — and has two table types: Copy-On-Write (rewrites affected data files on each write, optimized for read-heavy workloads) and Merge-On-Read (appends delta log files on write and merges on read, optimized for write-heavy workloads). Hudi's record-level indexing (Bloom filter, bucket, or HBase) enables sub-second point lookup latency on large datasets, which neither Delta Lake nor Iceberg supports natively without clustering. The right format depends on the dominant workload: CDC ingestion (Hudi), multi-engine federation (Iceberg), or deep Databricks/Spark ecosystem integration (Delta Lake).

Why does the open table format choice create vendor lock-in even though the formats are open source?

Open table formats create vendor lock-in through two mechanisms: managed-service feature extensions and ecosystem coupling. Managed-service feature extensions are proprietary optimizations that the managed service adds on top of the open format specification. Databricks Delta Lake adds V-ORDER (a write-time optimization that reorders values within Parquet pages for faster Photon engine reads), liquid clustering (a dynamic file compaction strategy that uses Databricks-specific metadata), and Change Data Feed (a mechanism for tracking row-level changes that is not part of the open Delta Lake specification). Tables that use these extensions cannot be read outside the Databricks platform without conversion — specifically, a Delta Lake table with liquid clustering enabled requires a Databricks compute cluster to read correctly; a standard open-source Delta Lake reader will produce incorrect results or fail. Ecosystem coupling is subtler: a team that builds ETL pipelines using Databricks Notebooks, schedules them with Databricks Workflows, and monitors them with Databricks Unity Catalog lineage has built a data platform where the table format is theoretically portable but the operational tooling is entirely platform-specific. Migrating the tables to a different query engine costs relatively little; migrating the orchestration, access control, lineage tracking, and notebook-based transformation logic costs much more. The lock-in surface of a managed lakehouse platform should be inventoried at selection time: which features does the team plan to use, which are open-specification, and which are proprietary extensions that reduce portability? Teams that document this inventory at platform selection time can make deliberate decisions about which proprietary features are worth the reduced portability, rather than discovering the portability cost when a migration is forced upon them.

What is schema evolution in a data lakehouse and what makes some schema changes safe versus unsafe?

Schema evolution in a data lakehouse refers to changing the schema of a table (adding, removing, renaming, or retyping columns) while maintaining the ability to read both new data (written after the change) and old data (written before the change, accessible via time-travel queries). A schema change is backward-compatible if readers using the old schema can still read data written with the new schema; it is forward-compatible if readers using the new schema can still read data written with the old schema (required for time-travel). Adding a nullable column is backward-compatible (old readers ignore the column, new readers see null for rows written before the column existed) and forward-compatible (new readers reading old files see null for the absent column). Deleting a column is backward-compatible for forward reads (new writers simply don't write the column; old readers that requested the column receive null) but is forward-incompatible for time-travel (readers with the new schema, which doesn't include the deleted column, may or may not handle old files where the column exists — most Parquet readers handle this correctly by ignoring unexpected columns, but application code that assumes the deleted column doesn't exist may behave unexpectedly when reading pre-deletion snapshots). Renaming a column is the most format-dependent operation. In Delta Lake (without Databricks column mapping enabled), renaming a column is implemented as adding a new column with the new name and deleting the old column — this is a destructive change that breaks readers using the old name. In Apache Iceberg, each column has a stable integer ID (the field_id) in addition to its name; renaming a column changes only the name, not the ID, so readers that use column IDs for projection (Iceberg's native readers) continue to read the same physical column regardless of the rename. Application code that references columns by name (SQL queries, dataframes) must be updated, but the physical data and metadata remain intact. The schema evolution safety contract must be documented in the data lakehouse decision record: which schema changes require a compatibility review before execution, which require all downstream readers to be updated before the change is applied, and what the time-travel compatibility guarantee is for changes that are backward-compatible but forward-incompatible.

What is the small file problem in a data lakehouse and how does compaction affect query latency?

The small file problem arises because every write to an open table format table creates one or more Parquet files. A batch ETL job that writes 100MB of data per run, running hourly, creates 24 files per day. After six months, the table has approximately 4,300 files averaging 100MB each — this is a manageable file count. But a streaming ingestion job that writes every 5 minutes creates 288 files per day, and after six months has approximately 52,000 files. A selective query on a large table with 52,000 files must open and inspect the metadata (or footer) of potentially thousands of files to determine which contain data matching the query predicate. The time to open file metadata scales roughly linearly with the number of files, even if the actual data reading is fast. On S3, each file metadata open requires an HTTP request with approximately 5 to 20 milliseconds of latency; 52,000 files with an average 10ms metadata read latency adds 8.7 minutes of file listing overhead before any data is read. Compaction is the process of merging small files into larger ones to reduce the file count and file listing overhead. Delta Lake's OPTIMIZE command (or automatic compaction in Databricks) rewrites existing files into larger target file sizes (typically 128MB to 1GB). Iceberg's compaction (the RewriteDataFilesAction in Spark, or automated compaction in managed services) does the same. Hudi's inline compaction runs during the write job for Merge-On-Read tables, merging delta log files into base files. The file size target and compaction frequency are the primary levers for query latency: larger target file sizes reduce file count and metadata overhead but increase the memory required per executor to hold a single file during processing. A compaction policy that targets 256MB files with a 1-week compaction cadence on a table receiving 10GB per day will produce a table with approximately 1,400 files after 6 months — much more manageable than 52,000. But the compaction cadence is an operational decision that must be built into the pipeline infrastructure, not an automatic consequence of the format choice. The format choice determines which compaction mechanisms are available, what they cost in compute, and whether they can run concurrently with reads and writes or require an exclusive maintenance window.

The data lakehouse decision record: why the open table format you chose determines your query latency ceiling and your schema evolution complexity

July 10, 2026 · WhyChose

A data lakehouse is adopted during a sprint that is framed as "modernising our data platform." The team replaces an aging data warehouse or a collection of unmanaged Parquet files with a structured lakehouse architecture — object storage for raw data, an open table format for ACID transaction support and time-travel queries, a distributed compute engine for transformation. The decision that determines the most about what the team can do with the lakehouse for the next several years is the open table format selection: Delta Lake, Apache Iceberg, or Apache Hudi. This decision is almost never written down as a decision. It is treated as an implementation detail chosen by whichever senior data engineer leads the platform sprint, based on whichever format they used at their previous company.

The open table format choice is not an implementation detail. It determines the query latency achievable on large datasets (because the file layout model and the record-level indexing approach differ fundamentally across formats), the safety of schema evolution operations (because the formats use different mechanisms — column IDs versus column names, copy-on-write versus merge-on-read — that create different compatibility guarantees for historical data), the concurrent write behavior (because the optimistic concurrency control mechanisms create different failure modes under simultaneous write loads), and the migration cost out of the platform (because managed lakehouse services layer proprietary extensions on top of the open format specification that create coupling to the vendor's compute environment even when the underlying Parquet files are nominally portable). All four properties are set at format selection time and are expensive to change retroactively because changing the format requires rewriting every table and rebuilding the metadata layer.

The missing decision record is invisible as long as the platform works. A Delta Lake table that is queried from a single Databricks workspace performs well, because Databricks optimizes the full read path for its own format. The latency ceiling imposed by the file layout model is not visible until the team tries to achieve sub-second point lookup latency that requires record-level indexing the format doesn't natively support. The schema evolution risk is not visible until someone deletes a column and discovers that the ML team's time-travel training pipeline silently ingested corrupted feature values for months before the model regression surfaced. The vendor lock-in cost is not visible until an acquisition, a cloud migration, or a platform consolidation makes the migration cost suddenly concrete and urgent. By then, the decision is two or three years old and the person who made it may not be with the company anymore.

Two things that happen when the decision is not written down

The managed-service coupling trap: a $47,000 migration cost that was never in scope

A 50-person data engineering team at a B2B SaaS company adopted Databricks to replace their Redshift warehouse. The migration was driven by two requirements: the team needed to run Python-based machine learning transformations alongside SQL analytics (which Redshift did not support natively at their scale), and they needed a unified platform for batch and streaming workloads. The Databricks platform addressed both requirements. Over 18 months, the team migrated three years of production data into 40 Delta Lake tables in the Databricks Unity Catalog, built 85 transformation notebooks using PySpark, and scheduled all pipelines using Databricks Workflows. The platform worked well.

The format selection decision, made in the first week of the migration project, was recorded as a single Slack message: "Going with Delta Lake — it's what Databricks uses natively and there's more documentation." No one documented the proprietary feature set that the team planned to use, which features were part of the open Delta Lake specification versus Databricks extensions, or what the migration path out of the platform would look like. The team progressively adopted Databricks features as they encountered them: Change Data Feed for tracking row-level changes in three high-activity tables, liquid clustering on the largest event table (replacing manual partition management), and Unity Catalog for access control and lineage. Each adoption was a reasonable tactical decision; none was flagged as a decision that increased vendor coupling.

Eighteen months after the initial migration, the company was acquired. The acquirer ran their data platform on Snowflake and had a policy requiring all subsidiary data to be consolidated into the parent's platform within twelve months of acquisition. The platform migration scoping session produced the first formal inventory of what the 40 Delta Lake tables actually contained. The scoping revealed three tiers of migration complexity. Tier one: twenty-one tables using standard Delta Lake features (partitioning, basic schema evolution, time-travel). These could be migrated by converting Parquet files and generating Snowflake COPY INTO statements — estimated two weeks of engineering effort. Tier two: seven tables using Delta Lake Change Data Feed. CDF is a Databricks-specific mechanism that appends change rows to a separate _change_data directory alongside the main table data. Snowflake does not read CDF data natively; the CDC consumers reading these tables via the CDF API would need to be rewritten to use Snowflake Streams, a different change tracking mechanism with different semantics. Estimated six weeks. Tier three: twelve tables using liquid clustering and column mapping enabled. Liquid clustering is implemented using Databricks-specific metadata in the Delta log that instructs Databricks's Photon engine how to physically reorganize files during compaction. A standard Delta Lake reader (the open-source delta-rs library, for example) ignores the liquid clustering metadata and reads the underlying Parquet files, but returns data without the performance characteristics the clustering was designed to provide. More critically, column mapping — enabled on these tables because engineers had renamed columns after initial deployment — uses a Databricks extension to the Delta format that remaps column names to Parquet field names using a separate metadata layer. The open Delta Lake specification does not include column mapping. Reading these tables outside of Databricks produces column name errors: the Parquet files contain columns named col-1, col-2 (internal identifiers assigned when column mapping was enabled), and the mapping back to human-readable names exists only in the Databricks-managed Delta log metadata. These twelve tables required a full rewrite — export from Databricks as raw Parquet with explicit name remapping, schema reconstruction, validation, and reload into Snowflake. Three months of engineering work. Databricks compute costs for the export and rewrite jobs: $47,000.

The $47,000 and three months were not in anyone's projection when the acquisition closed, because no one had documented the coupling surface at platform selection time. The CDF and column mapping decisions were each individually reasonable; collectively, they made twelve tables non-portable without a complete rewrite. The scoping engineer who produced the migration estimate spent two weeks just determining which tables used which features — there was no inventory, no decision record, and no architecture document that enumerated the Databricks-specific features in use. The data platform had no document that answered the question "what would it cost to migrate to a different platform?" because the decision to evaluate that question was never made. That question, answered honestly at platform selection time, might have produced a different tier-three design choice — using only open-specification Delta Lake features, or choosing Iceberg specifically for its multi-engine portability guarantee.

The schema evolution gap: eight months of silently corrupted ML training data

A 30-person analytics team migrated from a monolithic Postgres reporting database to an Apache Iceberg lakehouse on Amazon S3, running Spark transformations on EMR. The Iceberg format was chosen for its multi-engine compatibility: the team planned to query the same tables from both Spark (for batch transformation) and Trino (for interactive analytics), and Iceberg's compatibility across engines was a documented advantage. The migration shipped on schedule. For the first year, the platform performed as expected.

The schema evolution policy was informal. The team had read that Iceberg supported schema evolution natively, and interpreted this as "we can evolve the schema freely because Iceberg handles it." Engineers added columns routinely as new data sources were integrated. Column renames — replacing abbreviated names like usr_id with user_id — were performed monthly via Iceberg's ALTER TABLE RENAME COLUMN command. Column deletions, for deprecated fields that no longer had upstream sources, were batched quarterly and executed in maintenance windows. No schema change required a review. No reader compatibility validation was performed before applying changes. No time-travel compatibility guarantee was documented.

Twelve months into the platform's operation, a senior data scientist added a ml_features struct column to the main events table. The struct contained eight fields: user engagement signals computed from the past 30 days of activity for each event's associated user — session count, feature usage vector, and retention probability scores from a separately-run model. The struct was used as the feature input for a new event prioritization model. The column was added via a standard Iceberg ALTER TABLE ADD COLUMN statement; the change deployed without review because the policy required no review for column additions.

Six months after the struct column was added, an engineer cleaning up deprecated analytics fields ran a quarterly column deletion. The deletion batch included two fields from the ml_features struct: ml_features.retention_probability and ml_features.feature_usage_v1. Both had been replaced by updated versions (ml_features.retention_score_v2 and ml_features.feature_usage_v2) three months earlier, and the old fields were confirmed unused in all current Spark transformation jobs. The Iceberg schema change executed successfully. The deletion was backward-compatible for forward reads — new data did not include the deleted fields, and Spark queries against the current table schema returned correct results.

Two months after the column deletion, the data science team's batch model retraining pipeline failed with a NullPointerException in the feature extraction step. The pipeline was not a real-time inference job; it was a weekly batch job that retrained the event prioritization model on 8 months of historical event data, using Iceberg's time-travel capability to query snapshots at specific dates and reconstruct the feature set as it existed at each point in history. The time-travel queries correctly returned the historical schema — Iceberg's snapshot isolation guarantees that a query against a snapshot at a given timestamp sees the schema that existed at that timestamp, including columns that have since been deleted. The ml_features struct in the historical snapshots included retention_probability and feature_usage_v1 with valid values.

The problem was in the feature extraction code. The code queried the events table with a time-travel parameter, then applied a Python transformation function that projected the ml_features struct into a flat feature array using the current schema's field list — the schema that no longer included retention_probability and feature_usage_v1. The Spark DataFrame returned by the time-travel query contained all eight historical struct fields (Spark correctly materialized the historical schema), but the Python transformation function was written against the post-deletion schema and silently dropped the two deleted fields from the feature array, treating them as absent. The result: for any training example derived from an event in the 6-month window before the column deletion, the feature array was missing two values, producing a feature vector with 6 elements instead of 8. Numeric features in Python arrays do not produce errors when elements are missing — the array is simply shorter. The feature array was passed to a scikit-learn model that expected exactly 8 features; scikit-learn raised no error because the model was being retrained (not predicting), and the training code used the first N samples to determine the feature count. If most rows had 6-element arrays (because most of the training data predated the column deletion), the model retrained on 6 features, silently ignoring the two fields that were present only in the most recent 2 months of data. The retrained model had 7% lower precision on a held-out test set compared to the previous version. The degradation was within the threshold for automatic model promotion. The degraded model deployed to production.

Root cause investigation took six days. The data science team initially attributed the regression to distribution shift in the training data. A data engineer spent two days ruling out pipeline failures in the feature computation jobs. The investigation reached the schema change history only after the data scientist noticed that the feature importance output of the retrained model ranked only six features (not eight) as having nonzero weights — at which point the team re-read the Iceberg schema change log and found the column deletion from two months earlier. The six-day investigation involved six engineers and the data science team lead. The model that had been running in production with degraded precision for two months was retracted and retrained. The schema evolution policy was updated — informally, in a Slack thread — to note that struct column deletions required a review. No audit trail existed for the original column deletion decision. No reader compatibility assessment had been performed. The time-travel compatibility guarantee had never been defined, so no one had identified the time-travel scenario as a risk when the deletion was reviewed. The review process had not existed at all.

Three structural properties that are set at open table format selection time

The file layout model and the query latency ceiling

All three major open table formats store data in Parquet files. The difference in query latency across formats is not a property of Parquet; it is a property of how the format manages file layout — the distribution of records across files, the sort order within files, and the metadata that allows query engines to skip files that cannot contain matching records. The file layout model sets the minimum achievable query latency for selective queries on large datasets: a query that filters for all events by a specific user on a specific date can be answered by reading a single file (if the table is sorted by user ID and date and the file statistics are maintained) or by scanning every file in the table (if the layout is random and no statistics exist). The latency difference is orders of magnitude at the scale where a lakehouse is typically deployed.

Apache Hudi implements record-level indexing as a first-class feature. The Bloom filter index maintains a per-file Bloom filter that records which record keys (typically a primary key like user_id or event_id) are stored in each file. A point lookup query for a specific record key can use the Bloom filter to eliminate files that cannot contain the key (a Bloom filter false-negative rate of zero means no file is incorrectly eliminated), reducing the scan from all files to a small set of candidate files. The HBase index stores the record-to-file mapping in an external HBase table, providing O(1) lookup time for any record key — at the cost of maintaining an HBase cluster and accepting the network round-trip for every lookup. The bucket index partitions records by a hash of the record key into a fixed set of buckets (files), enabling direct file identification for any point lookup without an external index. Hudi's record-level indexing makes it the natural choice for CDC ingestion workloads (where the incoming stream contains upserts and deletes that must be applied to specific records) and for operational data serving workloads that require sub-second point lookup latency. The tradeoff is write overhead: maintaining the index on every write adds latency and compute cost proportional to the index type (Bloom filter updates are cheap; HBase updates require a network call per record).

Apache Iceberg manages file layout through sort orders and partition specs. An Iceberg table's partition spec defines how data is physically organized into directories (partitions), and the sort order defines how records are ordered within each Parquet file. A table partitioned by date and sorted by user_id within each date partition will have efficient query performance for date-range queries (only files in the relevant date partitions are scanned) and for user-specific queries within a date range (file-level min/max statistics on the sort key allow skipping files that cannot contain the queried user_id). Iceberg's hidden partitioning avoids the partition spec evolution problem that plagued Hive-style partitioned tables: in a Hive table, the partition column must be physically present in the data (a year=2024/month=07 directory structure requires a year and month column in the data or a synthetic partition column), making partition spec changes a schema migration. In Iceberg, the partition spec is separate from the data schema; the partition spec can be evolved independently, and Iceberg manages the metadata that maps between partition values and physical file locations. This connects to the database migration strategy decision record: partition spec evolution in a Hive-style table requires a data migration to rewrite all existing files into the new partition structure; in Iceberg, the evolution is metadata-only for new data, with existing data remaining in the old partition structure until explicitly compacted. The decision record must specify the partition spec migration policy — whether the team will rewrite historical data to match new partition specs, or maintain dual-partition metadata for tables that have evolved their spec.

Delta Lake's file layout management has diverged between the open-specification and the Databricks-managed implementations. The open specification supports OPTIMIZE (file compaction to target file sizes) and ZORDER BY (a space-filling curve that co-locates multi-column query predicates). Databricks's managed Delta Lake adds liquid clustering, which dynamically reorganizes files based on actual query patterns observed in the Databricks query history — a feedback loop that optimizes file layout for the most frequent queries automatically. Liquid clustering is effective for read performance but, as noted in the vendor lock-in story, is stored in Databricks-specific metadata that creates coupling to the Databricks platform. The file layout policy for a Delta Lake table must specify which layout management features the team will use, whether those features are in the open specification, and what the maintenance cadence for compaction will be. A Delta Lake table that is never compacted will accumulate small files from every incremental write job and experience query latency degradation over time as the file count grows. This connects to the data pipeline decision record: the compaction job is itself a pipeline component that must be scheduled, monitored for failures, sized appropriately for the table's data volume, and coordinated with read and write jobs to avoid conflicting with in-flight queries. A compaction job and a write job attempting to operate on the same files simultaneously will produce a conflict in Delta Lake's transaction log, causing one of the jobs to fail and retry. The compaction scheduling policy — when compaction runs, how often, and what the conflict resolution behavior is — must be part of the lakehouse decision record.

The schema evolution model and the reader compatibility surface

Each open table format has a different internal mechanism for tracking column identity, and the column identity mechanism determines which schema changes are safe, which are lossy, and which require all readers to be updated before the change is applied. The schema evolution model is the most format-specific property, and it is also the most consequential for teams with diverse reader populations — multiple query engines, multiple versions of transformation code, and time-travel workloads that query historical snapshots.

Apache Iceberg uses integer column IDs as the stable column identity. Every column in an Iceberg table has a numeric field_id assigned at creation time (e.g., field_id 1 for event_id, field_id 2 for user_id). The Parquet files store data using field IDs in the Parquet footer metadata. When a column is renamed — ALTER TABLE events RENAME COLUMN usr_id TO user_id — Iceberg changes the name in the schema metadata but keeps the same field_id. All existing Parquet files continue to work correctly because they reference the column by field_id, not by name. Readers that use Iceberg's native projection by field_id (Spark, Trino, Flink using Iceberg connectors) read the renamed column correctly without any file rewrite. Application code that references the column by name in SQL queries must be updated, but the physical data is intact. This is a genuinely safe rename: no data is rewritten, no historical snapshots are affected, no readers that have already executed are broken (because their result sets were materialized before the rename). The only breakage is in application code that references the old name going forward. Column deletion in Iceberg removes the column from the schema metadata. For new data files written after the deletion, the column is absent. For historical data files (pre-deletion snapshots), the column is present in the Parquet footer. When an Iceberg reader queries a post-deletion snapshot, it ignores the absent column (the schema says the column doesn't exist, so the reader does not project it). When an Iceberg reader queries a pre-deletion snapshot via time-travel, it reads the column from the historical files — Iceberg's snapshot isolation returns the schema as it existed at the queried snapshot's creation time. Application code that queries a pre-deletion snapshot and then projects the result using the post-deletion schema (assuming the deleted column doesn't exist) will receive the historical data correctly but may fail to handle it correctly if the code was written assuming the column is absent.

Delta Lake uses column names as the column identity by default. Renaming a column in Delta Lake (without column mapping enabled) requires adding a new column with the new name, backfilling the values, and dropping the old column — a destructive operation that rewrites the data and creates an incompatible break for readers using the old name. With column mapping enabled (a Databricks extension), Delta Lake can track column renames via a metadata layer similar to Iceberg's field IDs, but this extension is not part of the open Delta Lake specification and creates the portability issue described in the first story. The schema evolution model for Delta Lake must be specified in the decision record: will the team use column mapping (accepting the vendor coupling), avoid column renames (accepting a naming convention constraint), or execute renames as add-then-delete (accepting the operational cost and breakage window)? This connects to the data governance decision record: a data governance framework that includes a data dictionary (mapping logical column names to their physical identities in the storage layer) must accommodate the format's column identity model. A governance tool that identifies columns by name will break when a Delta Lake column is renamed via add-then-delete; a governance tool that uses Iceberg field IDs will correctly track the column through renames but requires the governance layer to be schema-format-aware.

Hudi's schema evolution support depends on the table type and the record format. Copy-On-Write tables use Avro as the internal record format by default, and schema evolution follows Avro's compatibility rules: adding a nullable field is backward-compatible; deleting a field requires marking it as optional (if it was required) in all existing files, which means a schema migration; renaming a field requires the Avro alias mechanism or a full table rewrite. Merge-On-Read tables handle schema evolution differently for base files (the full data files) versus delta log files (the incremental change files) — a schema change that is applied to new delta log files may produce a mismatch during merge operations if the base files contain the old schema and the delta log files contain the new schema. Hudi's schemaEvoLution configuration controls whether schema mismatches are resolved automatically (by promoting types and filling defaults) or fail-fast (by raising an error when the schemas are incompatible). The configuration must be set explicitly and documented in the decision record, because the default behavior in different Hudi versions has changed, and an undocumented configuration produces whatever the default is for the version deployed — which may not be the behavior the team intends.

The reader compatibility surface is the set of all systems and code that read the lakehouse tables and could be broken by a schema change. The surface must be inventoried as part of the schema evolution policy: which tables are read by which query engines, which engines use Iceberg's native projection (safe for renames), which use column-name-based projection (break on renames), and which run time-travel queries (affected by forward-incompatible deletions). This connects to the feature store decision record: if the lakehouse tables are also used as the offline feature store for ML model training — which is a common architectural choice, since the lakehouse already holds the historical data that features are computed from — then time-travel queries are not an edge case; they are the primary read pattern for model retraining. Every schema change on a table used for ML training must be evaluated for time-travel compatibility before it is applied. The evaluation must consider not just the format's compatibility guarantee but the application code that performs the time-travel query: does the feature extraction code use Iceberg's native projection (which respects the historical schema) or does it apply a post-extraction transformation using the current schema (which may be incompatible with the historical data)?

The concurrent write model and the optimistic concurrency surface

A data lakehouse is a multi-writer environment. Multiple ETL jobs, streaming ingestion pipelines, and incremental update processes write to the same tables simultaneously. The open table format's concurrency control mechanism determines what happens when two write jobs attempt to commit changes to the same table at the same time: whether conflicts are detected, which job wins, whether the losing job must retry or fails permanently, and what the atomicity guarantee is for partial writes. The concurrent write model sets the orchestration requirements for the data pipeline: a format that fails-fast on conflicts requires the orchestrator to implement retry logic; a format that silently allows last-write-wins produces data loss without any error signal.

Delta Lake uses optimistic concurrency control with a transaction log stored in the _delta_log/ directory. Each write job reads the current log version (the highest-numbered JSON file in the log directory), writes its data files to the table's storage location, and then attempts to write a new log entry that atomically records the set of files added and removed by the write. The atomicity is guaranteed by object storage's conditional write semantics: S3 does not natively support conditional writes (write-if-not-exists), so Delta Lake implements optimistic concurrency by attempting to write a log entry at a specific version number and treating a conflict (another writer already wrote that version) as a transaction conflict. The conflicting writer must re-read the log, apply its write logic to the updated state, and re-attempt the commit. If two writers conflict on non-overlapping partitions (writer A writes to date=2026-07-01, writer B writes to date=2026-07-02), Delta Lake can determine that the conflict is benign and allow both commits to succeed. If the writes overlap (both writers modify rows in the same partition), Delta Lake raises a ConcurrentModificationException and the losing writer's transaction must be retried from the read step. The retry behavior is not automatic in most Spark job configurations — the exception propagates to the Spark driver, the job fails, and the orchestrator (Airflow, Databricks Workflows, or similar) must restart the job. Without explicit retry logic in the orchestrator, a concurrent write conflict produces a permanent job failure that requires manual intervention. This connects to the CI/CD pipeline decision record: the pipeline that manages data transformation jobs is itself a deployment artifact with a version history; the retry policy for concurrent write conflicts must be part of the pipeline specification, not an undocumented assumption about the job failing occasionally and being manually restarted.

Apache Iceberg implements catalog-level locking to serialize table commits. The locking mechanism depends on the catalog implementation: a Hive Metastore catalog uses a distributed lock in the Hive database; a REST catalog (the emerging standard for Iceberg deployments) uses an optimistic concurrency mechanism implemented by the catalog server; a JDBC catalog uses database-level row locks. The catalog lock ensures that only one writer can commit a new snapshot at a time, eliminating the conditional-write race condition that Delta Lake manages via the transaction log. The tradeoff is that the catalog becomes a potential bottleneck for high-throughput concurrent write workloads: if 20 Spark jobs are writing to the same table simultaneously, they must serialize their commits through the catalog lock, and the commit throughput is bounded by the catalog server's capacity to handle lock acquisitions and releases. For most batch workloads, catalog-level locking is not a bottleneck — commits are infrequent relative to the data processing time, and the serialization overhead is a small fraction of the total job duration. For streaming ingestion workloads that commit every 5 to 30 seconds, catalog-level locking can become a throughput constraint, and the catalog server's horizontal scalability must be verified against the expected commit rate before deployment. This connects to the container orchestration decision record: the Iceberg catalog server is a stateful service that must be deployed, scaled, and operated alongside the lakehouse infrastructure; its availability is a dependency for every write to any Iceberg table, making it a critical availability dependency that must be included in the infrastructure reliability model.

Apache Hudi implements concurrent write support through configurable lock providers. The default Hudi configuration is single-writer mode: concurrent writes to the same table are not supported, and simultaneous write jobs will corrupt the table's state because both writers will independently modify the same commit metadata files without coordination. Multi-writer mode requires configuring a lock provider — Zookeeper, HBase, AWS DynamoDB, or a filesystem-based lock — and enabling the hoodie.write.concurrency.mode=OPTIMISTIC_CONCURRENCY_CONTROL setting. Without explicit configuration review, a team that runs two Hudi write jobs against the same table simultaneously may not encounter an error — the jobs will both complete successfully — but the table metadata may contain inconsistent commit history that produces incorrect results on subsequent reads or compaction operations. The absence of an error signal makes Hudi's single-writer mode a silent data corruption risk in multi-writer pipelines, which is the most dangerous failure mode in a data platform: the data is wrong, but no job failed. The concurrent write model must be explicitly documented in the decision record, including whether multi-writer mode is enabled, which lock provider is configured, and what the maximum number of simultaneous writers the lock provider is sized to support. This connects to the data pipeline decision record: the lock provider is a shared infrastructure dependency for all concurrent write jobs; its failure (Zookeeper cluster outage, DynamoDB rate limit hit) causes all write jobs to block, making it a single point of failure that must be accounted for in the pipeline reliability model.

Five sections the data lakehouse decision record should address

1. Open table format selection and managed-service dependency

Document the open table format selection (Delta Lake, Apache Iceberg, or Apache Hudi) with explicit rationale for the choice over each alternative. The rationale should address the dominant workload that drove the selection: CDC ingestion and point-lookup workloads favor Hudi's record-level indexing; multi-engine query federation favors Iceberg's broad compatibility; deep Spark and Databricks integration favors Delta Lake. If the selection was driven by the managed service platform (Databricks defaults to Delta Lake, AWS Glue and EMR default to Iceberg in their most recent configurations), document that explicitly, because the platform dependency is then a shared dependency between the lakehouse choice and the managed service choice.

Document the managed-service feature inventory: which features of the managed lakehouse platform are in use, which are part of the open format specification, and which are platform-specific extensions that reduce portability. For Delta Lake on Databricks, the inventory must specifically enumerate whether Change Data Feed is in use, whether column mapping is enabled on any tables, and whether liquid clustering has been applied. For Iceberg on AWS Glue, whether any Glue-specific catalog features are in use that differ from the open Iceberg REST catalog specification. For Hudi on Databricks or EMR, which index type is configured and whether the index storage (HBase, DynamoDB) creates a dependency beyond the object storage layer. The inventory should be updated when new platform features are adopted — the lock-in surface grows incrementally with each new feature adoption, and the cumulative impact is only visible if the inventory is maintained. This connects to the data warehouse decision record: if the lakehouse coexists with a traditional warehouse (a common hybrid architecture during migration), the decision record must specify the data flow boundary between them — which data resides in the warehouse, which in the lakehouse, and what the mechanism is for queries that need to join across both. The boundary definition determines which workloads can benefit from the lakehouse's open format portability and which are locked into the warehouse anyway.

Document the migration exit strategy. A migration exit strategy specifies what it would cost, in engineering time and compute cost, to migrate the lakehouse to a different platform — a different managed service, a different open table format, or a self-managed deployment. The exit strategy is not a prediction that the migration will happen; it is documentation of what the current architecture implies for future flexibility. A lakehouse that uses only open-specification features has a migration path that involves converting Parquet files and regenerating metadata — a moderate cost that scales with data volume. A lakehouse that uses proprietary managed-service features has a migration path that requires feature-by-feature replacement of those features in the target platform, which may not have equivalent capabilities. Documenting the exit strategy at platform selection time reveals whether the team is making an implicit bet on the managed service's continued viability, pricing stability, and feature compatibility — and allows the team to make that bet explicitly rather than discovering it when the bet needs to be unwound.

2. File layout policy and compaction strategy

Document the partition spec (how the table's data is physically organized into directories), the sort order within partitions, the target file size for compaction, and the compaction cadence. The partition spec should be designed against the dominant query patterns identified before the table is created — not after the table is built and performance problems surface. For an events table queried primarily by date range and user ID, a partition by date with a sort order by user_id within each partition allows date-range queries to read only the relevant date partitions and allows user-specific queries within a date range to skip files that don't contain the queried user_id (using Parquet column statistics). Document the query patterns that the partition spec is optimized for and explicitly note the query patterns that the partition spec is not optimized for — cross-date user aggregations, for example, will always require a full scan of all date partitions regardless of the partition spec.

Document the compaction job specification: the tool used (Spark, a managed service compaction job, or a format-specific tool like Hudi's HoodieCompactor), the target file size, the frequency, the scheduling mechanism, and the conflict resolution policy when a compaction job overlaps with an active write job. For Iceberg, document whether the compaction is managed (AWS Glue Table Optimizer, Databricks Liquid Clustering equivalent) or manual (a scheduled Spark job running the RewriteDataFilesAction). For Delta Lake, document whether the OPTIMIZE command is run manually, on a schedule, or automatically enabled via Databricks auto-optimization. For Hudi, document whether inline compaction is enabled for Merge-On-Read tables and what the compaction threshold is (the ratio of delta log file size to base file size that triggers a compaction). The compaction specification must include the operational runbook: how does the on-call engineer identify when compaction is behind schedule (file count metric, query latency metric, or explicit compaction lag metric)? What is the procedure for manually running compaction to catch up? This connects to the observability platform decision record: file count per table, average file size per partition, and compaction job latency are the key operational metrics for a data lakehouse; they must be instrumented and alarmed in the observability platform rather than checked manually when query performance degrades.

3. Schema evolution policy and reader compatibility contract

Document the schema evolution policy: which schema change types are permitted without a review (adding nullable columns, widening numeric types), which require a compatibility review before execution (deleting columns, renaming columns in Delta Lake without column mapping, changing column types in ways that affect precision), and which are prohibited entirely (renaming required columns in Hudi COW tables without a rewrite, narrowing numeric types). The compatibility review process must specify what the review covers: which downstream readers reference the affected column, whether any readers use time-travel queries and would be affected by a forward-incompatible change, and whether the change requires application code updates in reader services before or after the schema change is applied.

Document the time-travel compatibility guarantee explicitly. The guarantee must specify what the team promises to readers executing time-travel queries: can a reader query any historical snapshot and receive the schema that existed at that snapshot's creation time, with the expectation that their query code will produce correct results? If so, the guarantee implies that application code reading historical snapshots must be written to handle schema variations across historical versions — the code cannot assume the current schema matches the schema of the queried snapshot. If the guarantee is weaker (time-travel is supported for data recovery but reader compatibility with old schemas is not guaranteed), the decision record must say so explicitly, and application code that performs time-travel queries must be audited against the schema change history to verify compatibility. This connects to the feature store decision record: ML model training pipelines that use time-travel to reconstruct historical feature sets are among the most sensitive readers to schema evolution — they query months of historical snapshots in a single job, and a schema incompatibility between the current application code and the historical schema produces silent feature corruption rather than an error. The compatibility contract for time-travel readers must be validated at schema change time, not retroactively when a model regression surfaces.

Document the column lifecycle management process: how are columns named when introduced (naming convention, review requirements), how are columns deprecated (adding a deprecation comment, notifying downstream readers, setting a deprecation timeline), and how are columns deleted (confirming all readers have migrated away from the column, documenting the deletion in a change log, verifying that no time-travel queries reference the column in the retention window). The column lifecycle process prevents the scenario where a column is deleted because it appears unused in current query logs, but is referenced in a weekly batch job that runs rarely enough to not appear in the 30-day query history window used for the usage analysis.

4. Concurrent write model and orchestration requirements

Document the concurrent write configuration: whether multi-writer mode is enabled (for Hudi), what the lock provider is, and what the maximum concurrent writer count the configuration supports. Document the conflict resolution behavior: does a write conflict produce an automatic retry (with what backoff and retry count limit), a permanent failure that requires manual restart, or a silent last-write-wins resolution that may produce incorrect results? The conflict resolution documentation must specify the behavior for each type of conflict: overlapping partition writes (both writers modify data in the same partition), non-overlapping partition writes (writers touch different partitions but the same table metadata), and concurrent schema changes (one writer is in the middle of a schema evolution while another is writing data with the old schema).

Document the orchestration requirements that follow from the concurrent write model. If write conflicts produce permanent failures (the common behavior for Delta Lake on non-Databricks Spark), the orchestration layer must implement job retry logic with a backoff strategy that reduces the probability of re-collision. The retry policy must specify the maximum retry count (to prevent infinite retry loops), the backoff delay between retries (exponential backoff with jitter is standard), and the alert threshold (if a job retries more than N times, alert the on-call engineer rather than continuing to retry silently). If the concurrent write model uses external locking (Iceberg with a catalog-level lock), the orchestration must handle lock acquisition timeouts — a write job that cannot acquire the catalog lock within a timeout period must either wait (blocking downstream jobs that depend on the write completing) or fail (allowing the orchestrator to reschedule with a delay). The timeout and failure behavior must be documented and tested before production deployment. This connects to the API versioning decision record: if the lakehouse tables are exposed via a data API (a query service that allows downstream applications to retrieve data without direct table access), the API's versioning model must accommodate the table's schema evolution — the API version guarantees must be consistent with the table's schema evolution policy, so that an API client using a specific version is not broken by a schema change that occurs within the same API version's lifetime.

5. Observability instrumentation for data quality and performance

Document the required instrumentation for each production lakehouse table. The minimum observability set includes: table health metrics (row count, file count, average file size, last write timestamp, last compaction timestamp), data quality metrics (null rate per column, out-of-range value rate for bounded columns, duplicate primary key rate for tables with unique key constraints), and query performance metrics (scan file count per query, bytes scanned per query, query duration at P50/P95/P99, number of files skipped by predicate pushdown). The table health metrics detect the small file problem before it becomes a performance crisis: a table where the file count is increasing faster than the row count is accumulating small files and should trigger a compaction job. The data quality metrics detect silent data corruption — a null rate spike on a column that was previously non-null indicates a bug in an upstream pipeline, not a schema change. The query performance metrics detect when the file layout is no longer matching the query patterns: a rising scan file count for a query that should benefit from predicate pushdown indicates that the partition spec or sort order is misaligned with how the query is actually filtering data.

Document the data freshness SLA per table and the monitoring mechanism for freshness violations. The freshness SLA specifies the maximum acceptable lag between when data is generated upstream and when it is available in the lakehouse for query. A table with a freshness SLA of 1 hour must have a monitoring job that alerts within 5 minutes of the SLA being violated — a 55-minute freshness gap at the alert trigger point gives the on-call engineer 5 minutes to diagnose and restart the pipeline before the SLA is breached. The monitoring mechanism must check the table's last-write timestamp against the current time and compare it against the freshness SLA. This is a simple check that is frequently missing from new lakehouse deployments because the table works at launch and freshness monitoring is deferred until "when we need it" — which is when the pipeline has been silently failing for 6 hours and a business stakeholder asks why the dashboard numbers are wrong. This connects to the data pipeline decision record: the freshness monitoring job is itself a pipeline component that must be scheduled, must have its own failure alerting, and must not be deployed on the same infrastructure as the pipeline being monitored (a monitoring job that runs on the same Spark cluster as the job it monitors will fail simultaneously with the monitored job, leaving the freshness violation undetected until a human notices).

Document the time-travel retention policy and the snapshot expiration configuration. Open table formats accumulate historical snapshots as data is written — each write creates a new snapshot, and old snapshots are retained until explicitly expired. An Iceberg table that is written hourly and retains all snapshots indefinitely will accumulate 8,760 snapshots per year; the metadata tree that tracks these snapshots grows proportionally, increasing the time required to resolve which files are live (needed) versus orphaned (can be deleted). The snapshot expiration configuration specifies the minimum age before a snapshot can be expired (protecting time-travel queries that need access to recent historical snapshots) and the maximum age after which snapshots are automatically expired (preventing unbounded metadata growth). The time-travel retention policy must be calibrated against the actual time-travel use cases: an ML retraining pipeline that needs to reconstruct features from 12 months ago requires a 12-month retention window; a data recovery use case that only needs the past 7 days requires a 7-day retention window. The retention window has a direct cost implication — longer retention means more historical Parquet files that cannot be garbage-collected, occupying object storage at the per-GB-month cost of the storage tier. The retention policy is a cost-utility tradeoff that should be made explicitly and documented.

What to do with the data lakehouse decision record

The data lakehouse decision record's primary audience is the data engineer who selects and configures the lakehouse platform, the data scientist who trains models on historical data from the lakehouse, and the platform engineer who operates the compute and catalog infrastructure. The data engineer needs the open table format selection rationale and the managed-service coupling inventory to make deliberate decisions about which proprietary features to adopt, rather than adopting every available platform feature because it solves a tactical problem without considering the cumulative portability cost. The data scientist needs the schema evolution policy and the time-travel compatibility guarantee to understand what assumptions their feature extraction code can safely make about the schema of historical snapshots — specifically, whether a column that exists in the current schema was always present in all historical snapshots, and whether a column that was present 6 months ago still exists in the current schema. The platform engineer needs the compaction policy, the concurrent write configuration, and the observability instrumentation to maintain the platform's performance and data quality over time as the data volume and write frequency grow beyond the initial deployment parameters.

If the data lakehouse decision record was not written when the platform was built, the most useful reconstruction starts with a format audit. For the primary lakehouse format in use, enumerate every format-specific feature that is enabled on each table: partition spec, sort order, column mapping, Change Data Feed, liquid clustering, inline compaction, record-level indexing. For each feature, determine whether it is part of the open format specification or a managed-service extension. For each managed-service extension, estimate the migration effort to replace it with an open-specification equivalent if the team needed to migrate to a different platform tomorrow. The sum of these migration estimates is the actual portability cost of the current architecture — the number that belongs in the decision record as the migration exit strategy. The ADR template provides the structure for documenting the current state as a decision — the chosen format, the rationale as best as it can be reconstructed, the consequences observed so far (including any performance or schema evolution issues that have surfaced), and the current understanding of the migration exit cost.

The WhyChose extractor can recover data lakehouse architecture decisions from ChatGPT or Claude conversations during the platform design and migration phases — the thread where the format selection was debated, the conversation where the first schema evolution policy was drafted informally, or the discussion where the compaction strategy was first proposed. These conversations contain the reasoning that was never formalized into documentation. The decisions never written down essay covers why data platform decisions are particularly likely to be missing from decision records: they are made by data engineers rather than software engineers, in tools (Jupyter notebooks, Slack threads, Databricks workspace comments) that are outside the engineering ADR conventions. The result is that the infrastructure decisions that govern the entire data platform — the query latency ceiling, the schema safety surface, the vendor lock-in cost — exist only in the memory of the people who made them, and surface as constraints when those people are unavailable and the constraints become binding.