The lakehouse: Iceberg, Delta, and tables that time-travel

LEARN · MODERN DATA ENGINEERING

A lakehouse table is more than a directory full of Parquet

A plain object-storage data lake is wonderfully simple: put Parquet files in S3, Azure Data Lake Storage, Google Cloud Storage, HDFS, or another filesystem-like service, then point query engines at them.

The simplicity starts to hurt as soon as multiple writers, evolving schemas, deletes, updates, reproducible queries, or concurrent jobs enter the picture.

Suppose an orders dataset contains these files:

orders/
├── part-00000.parquet
├── part-00001.parquet
└── part-00002.parquet

A query engine can read them, but the directory itself does not answer some important questions:

Which files constitute the current table?
What happens if a writer crashes after uploading half its files?
Can a reader see a consistent snapshot while another job is replacing data?
What schema should old files use after a column is added or renamed?
Which files belonged to the table yesterday?
How can an update atomically remove old rows and add replacements?
How do you find relevant files without recursively listing millions of objects?

This is the problem an open table format solves. Parquet remains the data-file format in many deployments, while a table format adds a transactional metadata layer above those files.

Apache Iceberg and Delta Lake are two prominent implementations of that idea. Both provide snapshot-style reads, transactional writes, schema evolution, metadata pruning, and time travel, but their metadata structures and operational models differ significantly.

The mental model: data files plus a versioned control plane

Think of the table as two layers.

The first layer contains bulk data:

data/
├── 00001.parquet
├── 00002.parquet
├── 00003.parquet
└── ...

The second layer describes which data files logically belong to a particular table state.

A commit therefore does not need to rewrite every data file. It can create new files, update metadata that references them, and atomically publish a new table state.

That distinction is what makes time travel practical.

Version N can point to one collection of files while version N+1 points to another. Files shared by both versions do not need to be copied. Historical reads reconstruct the older logical state from metadata.

The important principle is:

A snapshot is a logical view of files and metadata, not necessarily a physical copy of the dataset.

How Apache Iceberg represents a table

Iceberg organizes table state as a metadata tree.

At a high level, a snapshot leads to a manifest list; the manifest list identifies manifest files; those manifests identify data and delete files. Manifests also carry partition information and column statistics that query planners can use before touching the underlying Parquet data.

Conceptually:

table metadata
└── current snapshot
    └── manifest list
        ├── manifest A
        │   ├── data-file-001.parquet
        │   ├── data-file-002.parquet
        │   └── data-file-003.parquet
        └── manifest B
            ├── data-file-104.parquet
            └── data-file-105.parquet

That hierarchy matters enormously at scale.

The manifest list contains summary information that can eliminate whole manifests. Only surviving manifests need to be read, and those manifests contain file-level statistics that can eliminate individual data files. Iceberg therefore treats metadata itself rather like a multilevel index.

For a filter such as:

event_time >= 2026-09-01 AND event_time < 2026-09-02

the planner can potentially discard large parts of the table without listing every storage object and without opening every Parquet footer.

Schema evolution is based on field identity

An especially important Iceberg design choice is that columns have stable numeric field IDs.

A column is therefore not identified only by its display name or ordinal position. When a field is renamed, its ID remains unchanged. When a new field is added, it receives a new ID that is not reused for some deleted field.

That makes operations such as these metadata-oriented rather than file-rewrite-oriented:

Add a column.
Drop a column.
Rename a column.
Reorder columns.
Widen supported types.
Evolve nested structures.

The current Spark integration exposes those operations through normal ALTER TABLE syntax.

Partitioning is metadata, not application logic

Iceberg also supports hidden partitioning.

Instead of forcing applications to manufacture a visible event_date field merely because files are physically partitioned by date, a table can define a transform such as day(event_time). Writers produce the partition value and readers translate predicates automatically. The partition layout can later evolve without forcing every query to know the new physical organization.

That separation between logical schema and physical layout is one of Iceberg’s strongest architectural ideas.

How Delta Lake represents a table

Delta Lake takes a different route.

For a conventional filesystem-managed Delta table, the directory contains data files plus a _delta_log directory. Commits create numbered transaction-log versions, and checkpoints compact table state so readers do not need to replay an indefinitely growing sequence of JSON commits. Current Delta Kernel documentation describes checkpoints as an optimization for reconstructing table state faster.

A simplified local table might look like this:

orders/
├── _delta_log/
│   ├── 00000000000000000000.json
│   ├── 00000000000000000001.json
│   ├── 00000000000000000002.json
│   └── ...
├── part-00000-....snappy.parquet
├── part-00001-....snappy.parquet
└── ...

Each table version represents a consistent snapshot. The log records actions such as adding files, removing files, changing table metadata, and changing protocol capabilities.

Version 17 does not mean “the seventeenth copy of every Parquet file.” It means “the table state obtained after commit 17.”

That makes version-based time travel pleasantly concrete:

VERSION AS OF 17

Delta also has explicit reader and writer protocol capabilities. Modern Delta tables can enable table features that require compatible clients, so protocol compatibility becomes important in multi-engine environments.

Recent Delta releases additionally support catalog-managed tables, in which commit coordination can move from direct filesystem publication to a catalog-mediated model. The ordinary local example in this article intentionally stays with the simpler filesystem-managed model so the transaction log is easy to inspect.

The current stack for our runnable lab

As of September 2026, Apache Iceberg 1.11.0 is the current release and publishes runtime artifacts for Spark 4.1 with Scala 2.13. Delta Lake 4.4.0 supports Spark 4.2, 4.1, and 4.0 builds. PySpark 4.1.3 is the current 4.1 patch release.

Spark 4.2.0 exists, so why not use it here?

Because Iceberg 1.11.0’s maintained runtime matrix currently goes through Spark 4.1. Using Spark 4.1.3 gives us a recent Spark runtime that overlaps cleanly with both current table-format releases rather than mixing an Iceberg JAR compiled for a different Spark minor line.

You will need Python 3.10 or newer and Java 17 or newer. Iceberg 1.11 removed Java 11 support, and PySpark 4.1 requires Java 17+.

Create an isolated environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install pyspark==4.1.3
java -version

The table-format JVM packages will be fetched by Spark through Maven coordinates when each example starts.

We will run Iceberg and Delta in separate processes. That is not because they cannot coexist in a larger Spark deployment; it simply makes the tutorial easier to reproduce and lets each example use the canonical extensions and catalog configuration for its own format.

Lab 1: build an Iceberg table, evolve it, and travel backward

Create iceberg_lab.py:

from pathlib import Path
import shutil

from pyspark.sql import SparkSession


root = Path("iceberg_lab_data").resolve()

if root.exists():
    shutil.rmtree(root)

warehouse = (root / "warehouse").as_uri()

spark = (
    SparkSession.builder
    .appName("iceberg-schema-evolution-time-travel")
    .master("local[2]")
    .config(
        "spark.jars.packages",
        "org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0",
    )
    .config(
        "spark.sql.extensions",
        "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
    )
    .config(
        "spark.sql.catalog.local",
        "org.apache.iceberg.spark.SparkCatalog",
    )
    .config("spark.sql.catalog.local.type", "hadoop")
    .config("spark.sql.catalog.local.warehouse", warehouse)
    .config("spark.sql.shuffle.partitions", "2")
    .getOrCreate()
)

spark.sparkContext.setLogLevel("WARN")

spark.sql("CREATE NAMESPACE IF NOT EXISTS local.demo")

spark.sql(
    """
    CREATE TABLE local.demo.orders (
        order_id BIGINT,
        customer STRING,
        amount DOUBLE
    )
    USING iceberg
    """
)

spark.sql(
    """
    INSERT INTO local.demo.orders
    VALUES
        (1, 'Ada', 89.90),
        (2, 'Ben', 42.50),
        (3, 'Chloe', 215.00)
    """
)

snapshots = spark.sql(
    """
    SELECT committed_at, snapshot_id, operation
    FROM local.demo.orders.snapshots
    ORDER BY committed_at
    """
)

print("Snapshots after the first write:")
snapshots.show(truncate=False)

first_snapshot_id = snapshots.first()["snapshot_id"]

spark.sql(
    """
    ALTER TABLE local.demo.orders
    ADD COLUMN shipping_country STRING
    """
)

spark.sql(
    """
    INSERT INTO local.demo.orders
    VALUES
        (4, 'Dara', 125.50, 'DE'),
        (5, 'Eli', 77.25, 'FR')
    """
)

print("Current Iceberg table:")
spark.sql(
    """
    SELECT *
    FROM local.demo.orders
    ORDER BY order_id
    """
).show(truncate=False)

print(f"Iceberg table at snapshot {first_snapshot_id}:")
spark.sql(
    f"""
    SELECT *
    FROM local.demo.orders
    VERSION AS OF {first_snapshot_id}
    ORDER BY order_id
    """
).show(truncate=False)

print("All snapshots:")
spark.sql(
    """
    SELECT committed_at, snapshot_id, operation
    FROM local.demo.orders.snapshots
    ORDER BY committed_at
    """
).show(truncate=False)

spark.stop()

Run it:

python iceberg_lab.py

The current table should logically look like this:

+--------+--------+------+----------------+
|order_id|customer|amount|shipping_country|
+--------+--------+------+----------------+
|1       |Ada     |89.9  |NULL            |
|2       |Ben     |42.5  |NULL            |
|3       |Chloe   |215.0 |NULL            |
|4       |Dara    |125.5 |DE              |
|5       |Eli     |77.25 |FR              |
+--------+--------+------+----------------+

Old Parquet files did not suddenly acquire a new physical field. The evolved table schema says that shipping_country now exists, so rows from older files produce NULL for that optional field.

The historical query is more interesting. Iceberg’s current Spark documentation specifies that time travel by numeric snapshot ID uses the snapshot’s schema. Our first snapshot predates shipping_country, so its result has the original three-column schema rather than merely filling the later column with nulls.

Conceptually:

+--------+--------+------+
|order_id|customer|amount|
+--------+--------+------+
|1       |Ada     |89.9  |
|2       |Ben     |42.5  |
|3       |Chloe   |215.0 |
+--------+--------+------+

That detail is worth remembering: time travel can restore historical metadata semantics as well as historical rows.

Why we queried the metadata table first

Iceberg exposes metadata tables such as snapshots, files, manifests, history, and related views through Spark.

We did not invent a snapshot number. We retrieved the actual ID from:

local.demo.orders.snapshots

That is an important operational pattern. Snapshot IDs are identifiers generated by the table implementation; application code should discover them from metadata, tags, branches, audit records, or commit output rather than pretending that snapshots are simple monotonically increasing integers.

Iceberg supports SQL time travel with both VERSION AS OF and TIMESTAMP AS OF; versions can refer to numeric snapshot IDs as well as branches or tags.

Lab 2: perform the same experiment with Delta Lake

Now create delta_lab.py:

from pathlib import Path
import shutil

from pyspark.sql import SparkSession


root = Path("delta_lab_data").resolve()

if root.exists():
    shutil.rmtree(root)

table_path = (root / "orders").as_posix()

spark = (
    SparkSession.builder
    .appName("delta-schema-evolution-time-travel")
    .master("local[2]")
    .config(
        "spark.jars.packages",
        "io.delta:delta-spark_4.1_2.13:4.4.0",
    )
    .config(
        "spark.sql.extensions",
        "io.delta.sql.DeltaSparkSessionExtension",
    )
    .config(
        "spark.sql.catalog.spark_catalog",
        "org.apache.spark.sql.delta.catalog.DeltaCatalog",
    )
    .config("spark.sql.shuffle.partitions", "2")
    .getOrCreate()
)

spark.sparkContext.setLogLevel("WARN")

initial_orders = spark.createDataFrame(
    [
        (1, "Ada", 89.90),
        (2, "Ben", 42.50),
        (3, "Chloe", 215.00),
    ],
    "order_id LONG, customer STRING, amount DOUBLE",
)

initial_orders.write.format("delta").mode("overwrite").save(table_path)

first_history = spark.sql(
    f"""
    DESCRIBE HISTORY delta.`{table_path}`
    """
)

first_version = first_history.selectExpr(
    "min(version) AS version"
).first()["version"]

spark.sql(
    f"""
    ALTER TABLE delta.`{table_path}`
    ADD COLUMNS (shipping_country STRING)
    """
)

new_orders = spark.createDataFrame(
    [
        (4, "Dara", 125.50, "DE"),
        (5, "Eli", 77.25, "FR"),
    ],
    """
    order_id LONG,
    customer STRING,
    amount DOUBLE,
    shipping_country STRING
    """,
)

new_orders.write.format("delta").mode("append").save(table_path)

print("Current Delta table:")
spark.read.format("delta").load(table_path).orderBy(
    "order_id"
).show(truncate=False)

print(f"Delta table at version {first_version}:")
spark.sql(
    f"""
    SELECT *
    FROM delta.`{table_path}`
    VERSION AS OF {first_version}
    ORDER BY order_id
    """
).show(truncate=False)

print("Delta history:")
spark.sql(
    f"""
    DESCRIBE HISTORY delta.`{table_path}`
    """
).select(
    "version",
    "timestamp",
    "operation",
).orderBy(
    "version"
).show(
    truncate=False
)

spark.stop()

Run it:

python delta_lab.py

The sequence is deliberately almost identical to the Iceberg exercise:

Write three rows using the original schema.
Capture the historical table version.
Add shipping_country.
Append two rows using the evolved schema.
Read the latest table.
Query the original version.

Delta supports explicit ALTER TABLE ... ADD COLUMNS, and current documentation also supports per-operation automatic schema evolution in newer releases. Explicit DDL is preferable for this lab because you can see exactly when metadata changes rather than letting a write silently decide that evolution should occur.

Delta’s SQL syntax supports both VERSION AS OF and TIMESTAMP AS OF, while the DataFrame reader provides versionAsOf and timestampAsOf. DESCRIBE HISTORY exposes the commit versions you can feed into those historical reads.

Snapshot IDs and version numbers are not the same abstraction

The two labs look similar from SQL, but there is a meaningful conceptual difference.

Delta’s history is naturally expressed as transaction-log versions:

0
1
2
3
...

Every new commit advances the table version.

Iceberg snapshots instead carry snapshot IDs and participate in a richer metadata graph that can also be referenced by branches and tags. The current Spark API accepts a snapshot ID, branch, or tag in a version-based time-travel query.

So application code should not build a generic abstraction that assumes “version 12” means exactly the same thing in every table format.

A better cross-format abstraction is:

historical table reference
├── format
├── table identifier
├── snapshot/version reference
└── optional timestamp

Your catalog, data product, or orchestration layer can then translate that reference into the native mechanism of the target table format.

Schema evolution is where metadata earns its keep

Adding a nullable field appears deceptively simple.

Without a table-format layer, an application might discover that yesterday’s Parquet files contain three columns while today’s contain four. Different readers might infer different merged schemas depending on discovery order and configuration.

With a table format, the table itself owns the schema.

For our current Iceberg table, the metadata says:

order_id
customer
amount
shipping_country

Older files do not contain shipping_country, but their fields have known identities and the current schema can project a missing optional value as NULL.

Delta similarly records schema changes in table metadata rather than asking every reader to infer the union of arbitrary Parquet files. Current Delta documentation describes adding a column as a metadata-only operation in its Kernel alter-table implementation: no existing data files need to be rewritten simply to introduce the nullable field.

This is a major difference between “schema evolution” and “rewrite every byte.”

Automatic evolution deserves caution

Both ecosystems offer mechanisms that can automatically accommodate incoming schemas.

Iceberg’s Spark writer can accept new columns when the table is explicitly configured to allow schema changes and the writer enables schema merging.

Delta 4.3 and later supports operation-scoped schema evolution for writes, and its existing merge APIs can evolve compatible target schemas.

That convenience is powerful in controlled ingestion systems, but it should not become an excuse to abandon schema governance.

If any arbitrary producer can add fields automatically, a typo such as:

shipping_county

instead of:

shipping_country

can become persistent production metadata.

For important tables, treat automatic evolution as a governed capability rather than a blanket ingestion setting.

Time travel does not mean infinite history

A successful historical query today does not guarantee that the same version will remain readable forever.

Historical metadata and historical data files consume storage. Eventually most production systems clean them up.

Iceberg retention

Iceberg keeps old snapshots so they can provide isolation and time travel, then exposes maintenance procedures to expire snapshots that are no longer needed. Current Spark procedures include expire_snapshots; files still required by surviving snapshots are protected from deletion.

A production retention policy might keep a minimum number of snapshots while expiring sufficiently old history:

keep enough snapshots for:
- rollback
- reproducible jobs
- delayed consumers
- audit requirements
- incident investigation

Do not mechanically copy a retention period from another organization. Snapshot frequency matters: “seven days” could mean 168 snapshots for hourly commits or millions of snapshots for very high-frequency ingestion.

Delta retention

Delta time travel similarly depends on retaining both transaction-log information and the underlying data files needed by the requested version.

Current defaults document 30 days for log retention and seven days for deleted-file retention. Data files are not simply removed at the moment they disappear from the latest snapshot; VACUUM is the operation that makes old unreferenced data physically disappear. Once required files are vacuumed, the corresponding historical version cannot be reconstructed.

That leads to an operational rule worth putting on a runbook:

Time travel is a data-management feature, not a backup strategy.

A mistaken cleanup policy, compromised storage account, bucket deletion, or catastrophic object-store problem can destroy both current and historical versions if they live in the same failure domain.

The cherry on the cake: multi-petabyte planning from one node

The surprising part of lakehouse scale is not merely that Parquet can hold petabytes. Object storage has been able to hold enormous quantities of bytes for a long time.

The harder problem is discovering which tiny fraction of those bytes a query needs.

Apache Iceberg’s current performance documentation states that the format is used for tables containing tens of petabytes, and that even multi-petabyte tables can have their scans planned from a single node rather than requiring a distributed SQL job just to sift through all table metadata.

That sounds counterintuitive until you revisit the metadata tree:

snapshot
    ↓
manifest list
    ↓
selected manifests
    ↓
selected data files
    ↓
Parquet row groups

The planner first uses summary metadata to reject irrelevant manifests. It then uses manifest-level file statistics to reject irrelevant files. Only the resulting scan tasks reach the actual compute stage.

In other words, the answer to “How do you manage a table with an absurd number of files?” is not “list every file faster.”

It is stop requiring every query to discover every file in the first place.

Delta attacks the same broad scaling problem through its transaction log, checkpoints, statistics, and distributed metadata processing. Its current project documentation explicitly describes scalable metadata handling for petabyte-scale tables and very large file counts.

This is why metadata design is not bookkeeping around the real data. At large scale, metadata architecture becomes part of the query engine.

Iceberg versus Delta: the practical comparison

Both formats are capable foundations for serious analytical tables, so the useful question is rarely “Which one has time travel?” Both do.

The more useful comparison is where their abstractions fit your platform.

Concern	Apache Iceberg	Delta Lake
Historical identity	Snapshot IDs, timestamps, branches and tags	Transaction-log versions and timestamps
Core metadata shape	Metadata file → snapshots → manifest lists → manifests → files	Versioned transaction log plus checkpoints
Schema identity	Stable field IDs are central to the specification	Schema and protocol metadata are recorded in the transaction log
Partition model	Hidden partition transforms and partition evolution are first-class	Partitioning and newer clustering capabilities are table features
Spark time travel	`VERSION AS OF` / `TIMESTAMP AS OF`	`VERSION AS OF` / `TIMESTAMP AS OF`
Local lab storage	Hadoop catalog works well	Path-based filesystem-managed table works well
Historical cleanup	Snapshot expiration and orphan-file maintenance	Log retention plus `VACUUM` for unreferenced data
Catalog emphasis	Catalog is fundamental to table discovery and commits	Filesystem-managed and newer catalog-managed approaches exist
Branches/tags	Native snapshot references	Table history is primarily version-oriented

Do not choose solely by SQL syntax. At the query surface they deliberately look similar.

Choose based on the system around the table: catalogs, engines, write patterns, governance, streaming requirements, maintenance tooling, cloud integration, protocol compatibility, and the operational expertise already present in your team.

Do not forget the catalog

A table format answers “What constitutes this table state?” A catalog answers questions such as “Where is the table?”, “Who owns it?”, and often “Who is allowed to commit a new state?”

Our Iceberg lab uses a local Hadoop catalog because it needs no external services. Production systems commonly use catalog implementations backed by REST services, metastore systems, JDBC databases, or cloud catalog services.

The distinction matters because an Iceberg commit ultimately needs an atomic way to publish new table metadata. The storage files can already exist before that atomic metadata transition makes the new table state visible.

For Delta, our local example relies on filesystem-managed commits in _delta_log. Modern Delta development is also expanding catalog-managed commit coordination, so architects should no longer think of Delta as permanently synonymous with “only a directory of numbered JSON files.”

Avoid the small-files trap

Transactional commits make it easy to write frequently. That does not make millions of tiny Parquet files free.

A table with tiny files pays repeatedly:

More object-store requests.
More file-open overhead.
Larger metadata structures.
More scan tasks.
More compaction work.
More maintenance pressure.

Iceberg’s current Spark maintenance procedures provide rewrite_data_files for consolidating small data files, and the documentation explicitly notes that increasing file counts increases metadata stored in manifests.

Delta deployments have the same physical reality: a transaction log cannot make a five-kilobyte Parquet file efficient to scan merely because the commit that referenced it was atomic.

Your ingestion architecture therefore needs two scales:

commit frequency != ideal data-file size

You may commit frequently for freshness while periodically compacting files into a layout suitable for analytical scans.

Time travel is also a debugging primitive

Historical reads are often introduced as a rollback feature, but that understates their value.

Imagine a dashboard is correct at 09:00 and wrong at 10:00.

Without table history, debugging starts with reconstructing what data might have existed at 09:00.

With versioned tables, you can instead compare snapshots:

09:00 snapshot
vs.
10:00 snapshot

Then narrow the problem to:

New rows.
Deleted rows.
Changed metadata.
A schema transition.
A faulty upstream overwrite.
A transformation that consumed the wrong input version.

Historical reads also make reproducible data pipelines easier. A downstream job can record the exact input snapshot/version it consumed instead of merely recording a path that changes underneath it.

For production lineage, consider persisting information like:

{
  "dataset": "commerce.orders",
  "format": "iceberg",
  "snapshot_reference": "7429187349821734",
  "pipeline_run": "orders-daily-2026-09-04"
}

or:

{
  "dataset": "commerce.orders",
  "format": "delta",
  "version": 1842,
  "pipeline_run": "orders-daily-2026-09-04"
}

That tiny piece of lineage can turn an otherwise vague incident report into a reproducible query.

What to inspect when a lakehouse query is slow

Once you understand the metadata layer, performance diagnosis changes.

Do not begin only with executor CPU graphs. Ask whether the table is generating an unnecessarily large scan plan.

For Iceberg, inspect:

Snapshot history.
Number and size of manifests.
Number and size of data files.
Partition evolution.
Column statistics.
Whether filters can prune manifests and files.
Whether small files need compaction.

For Delta, inspect:

Table history.
Number and size of active files.
Checkpoint behavior.
Partitioning or clustering strategy.
Available data-skipping statistics.
Whether old files and log history are being maintained according to policy.

A query that scans ten terabytes because its predicates cannot prune files will not be fixed by shaving a few milliseconds from Python startup.

The architectural lesson

The major innovation behind modern lakehouse tables is not merely adding an UPDATE command to object storage.

It is replacing an ambiguous bag of files with an explicit, transactional, versioned model of table state.

Once you have that model, several capabilities emerge from the same foundation:

Readers can obtain snapshot isolation.
Writers can publish atomic changes.
Schemas can evolve without casually corrupting old data.
Query planners can prune metadata before touching bulk files.
Historical versions can be queried reproducibly.
Maintenance jobs can reason about which files are live, obsolete, or orphaned.

Iceberg expresses that model through snapshots, manifest lists, manifests, stable field identities, and catalog-coordinated metadata.

Delta expresses it through a versioned transaction log, table metadata, checkpoints, protocol features, and an increasingly broad ecosystem of filesystem- and catalog-managed integrations.

The next useful step is not to memorize another comparison chart. Run both labs, inspect the directories they create, add a second schema change, append another commit, and deliberately query each historical state. Then open the metadata tables or transaction history and trace exactly how your five tiny rows became a transactional table.

Build the examples, break them on purpose, time-travel back to the good state, and make metadata—not just Parquet—the next thing you learn to read.