How Iceberg snapshots, branches, and time travel work
Trace Iceberg metadata from atomic snapshots through named branches, divergent lineages, time-travel reads, retention, and file cleanup.
The whole story in 6 lines
See how Iceberg’s metadata tree, atomic commits, named references, divergent lineage, historical reads, and retention work together.
- Readers resolve a table reference to metadata, then follow snapshots and manifests to the exact data files for that state.
- An Iceberg commit builds candidate metadata separately and changes the live table only through an atomic pointer update.
- A branch is a named snapshot reference, so creating one copies metadata pointers rather than table data.
- Main and development branches can diverge while continuing to share unchanged manifests and data files.
- A time-travel read resolves a concrete snapshot before planning manifests and files, preserving one consistent table state.
- Time travel lasts only while snapshots remain reachable, and retention prunes metadata before orphaned files are deleted.
Setup
Welcome to Iceberg time travel. Think of a table that remembers every version of itself, like a document with infinite undo history. That is what Iceberg gives you.
A snapshot is an immutable description of the entire table state at one moment in time. Every committed write creates a new snapshot without touching the old one.
A manifest is a file that lists which data files belong to a snapshot. It is like a table of contents for one chapter of the table.
The catalog is the single entry point. It stores one atomic pointer to the current metadata file for each registered table name.
A branch is a named pointer to a snapshot. Main always points at the latest production state. Dev branches let you experiment without affecting main.
Now let us start with the most fundamental idea: how the metadata tree connects all these pieces together.
Metadata Tree
Here is the core question this stage answers: how does a reader find the actual data files for a table. The answer starts at the catalog, which holds just one pointer to the current metadata file.
That metadata file carries the table schema plus the full snapshot log. Think of it as the spine of a book that connects every chapter.
The active snapshot is the frozen picture we learned about in the Setup. It points at the table state at one exact moment.
The snapshot opens its manifest list, which fans out into individual manifests. Readers never discover data files directly. They resolve a ref, load the current metadata file, then walk snapshots and manifests.
Each manifest lists concrete data files on storage. Readers never discover data files directly. They resolve a ref, load the current metadata file, then walk snapshots and manifests.
That is the metadata tree in one picture: catalog to metadata to snapshot to manifests to files. No file is ever mutated in place. Next, we will see how a new write publishes itself through this chain.
Atomic Commit
A new write begins by reading the current snapshot and planning changes against that exact base. The live table remains untouched while candidate files and metadata are prepared.
New data files are staged on storage without touching the live head. Think of this as writing a draft next to the published book.
The writer builds a candidate snapshot that references the new files plus any unchanged manifests from the parent. This is the new chapter being assembled.
The stage has reached the decision that determines its next state. What changes atomically when an Iceberg commit succeeds?
Now comes the critical moment. The catalog attempts one atomic pointer swap, a compare-and-swap. If no other writer changed the pointer first, this commit wins.
The loser finds a stale base and must retry from the new head. That retry loop is the price of lock-free commits. Next, let us see how branches give different writers their own ref to advance.
Branch Refs
★ If you remember one thing · Creating an Iceberg branch adds a movable reference to an existing snapshot while unchanged files remain shared.
We just saw how a commit uses the atomic swap from Stage 2. But what if you want to experiment without affecting the production table. That is what branches are for. This detail matters because it determines the state handed to the next part of the system.
Main is a movable ref aimed at the production head, while a tag pins one exact snapshot. Both names point into the same immutable lineage rather than wrapping copied table data.
The stage has reached the decision that determines its next state. What does an Iceberg branch name point to?
The dev ref now attaches directly to the chosen snapshot. Both refs still reach the same physical file objects, so creating this branch added a pointer without duplicating unchanged table data.
Switch the Base control through both choices. Watch the dev arrow jump between the historical snapshot and current head while the shared file objects remain exactly where they are.
A new dev commit creates another immutable snapshot, then only the dev ref advances. Main and the audit tag stay put, which sets up the next stage: two lineages can diverge while still sharing files.
Divergent Lineage
Main and development begin from the same base snapshot, then commit independently. Their references can diverge while unchanged manifests and data files remain shared.
Main lands new commits and advances its ref. Each commit uses the atomic swap we learned about in Stage 2, but only main's pointer moves.
Dev writes its own changes independently. Main and dev can expose different table views while still referencing much of the same underlying storage.
Notice the shared files panel. Files from the base snapshot are still referenced by both branches until one of them rewrites or deletes them.
Both heads now expose entirely different table states, even though they share much of the same underlying storage. This is the power of immutable metadata plus lightweight refs.
After divergence, main cannot fast-forward to an unrelated head. A cherry-pick creates a new single-parent snapshot on main instead, because core Iceberg history is not a Git-style two-parent merge graph. Next, we will see how readers choose a view.
Time Travel Reads
We have seen how branches diverge in Stage 4. Now a reader arrives with a SQL query. How does it know which version of the table to see. That depends on the selector it provides.
Each gives a different way to name the exact table state you want. All three collapse to one concrete snapshot before the scan begins.
The resolved snapshot becomes the pinned read view. Even if main advances while this query runs, the read stays locked to the snapshot chosen at the start. That is the consistency guarantee.
Iceberg loads only the manifests reachable from that snapshot. Remember the manifest list from Stage 1. The same fanout applies here, but scoped to the resolved snapshot.
Partition pruning narrows the scan to only the matching data files. Files outside the query predicate are skipped entirely.
Returned rows always match the resolved snapshot. That one-snapshot-at-a-time rule is why time travel queries are consistent. Now let us step back and see what happens when old snapshots need to be cleaned up.
Retention Window
We have built up snapshots across all the previous stages. But snapshots accumulate forever unless something cleans them up. That is what the retention window does.
Live refs define what stays reachable. Main and dev preserve their paths, while an audit tag can keep an older snapshot alive even when it falls outside the ordinary age window.
The ordinary policy keeps the newest snapshots, but reachability wins over age. The older tagged snapshot remains live while an unreferenced predecessor becomes eligible for expiry.
Expired snapshots and manifest lists are dropped from metadata first. This is the same metadata tree from Stage 1, but now parts of it are being pruned instead of grown.
Only after metadata releases a file from every reachable snapshot does that orphan become safe to delete. A file still used by the audit tag remains in place, which prevents accidental data loss.
The table stays compact, but time travel now exists only inside the retained window. That tradeoff between history depth and storage cost is the final piece of the Iceberg puzzle. Now let us step back and see the whole picture together.
Recap
We started with the metadata tree: catalog to metadata to snapshot to manifests to files. That chain is the foundation everything else builds on.
Then we learned that writes never touch the live head. They build a candidate off to the side and publish with one atomic swap. That is how Iceberg stays consistent without locks.
Branches are named pointers into the snapshot chain. Creating one changes metadata and does not copy the unchanged data files already reachable from its starting snapshot.
When branches advance independently, they can diverge while still sharing files from their common ancestry. That reuse is what makes branches practical at scale.
Time travel reads pin to one snapshot before scanning. Branch names, snapshot ids, and timestamps all collapse to one frozen view. That is the consistency guarantee.
Retention cleans up old snapshots in two phases: metadata first, then orphan files. It keeps the table compact while preserving a configurable window of history.
All six concepts now connect. Immutable metadata makes the atomic pointer swap possible. Named refs let histories diverge, pinned reads select one state, and reachability-based retention bounds how long that state survives.
The whole story in 6 lines
See how Iceberg’s metadata tree, atomic commits, named references, divergent lineage, historical reads, and retention work together.
- Readers resolve a table reference to metadata, then follow snapshots and manifests to the exact data files for that state.
- An Iceberg commit builds candidate metadata separately and changes the live table only through an atomic pointer update.
- A branch is a named snapshot reference, so creating one copies metadata pointers rather than table data.
- Main and development branches can diverge while continuing to share unchanged manifests and data files.
- A time-travel read resolves a concrete snapshot before planning manifests and files, preserving one consistent table state.
- Time travel lasts only while snapshots remain reachable, and retention prunes metadata before orphaned files are deleted.