Apache Iceberg Architecture Explained

Apache Iceberg is an open table format designed for high-performance analytics running on object storage. By decoupling table state from directory file layouts, Iceberg brings ACID transactions, schema evolution, and time travel to object stores like S3, ADLS, and GCS.

At the structural level, a table catalog tracks the current metadata file pointer, which leads down an immutable tree to manifest lists, manifest files, and underlying data objects. This design delivers complete engine interoperability, but it also creates specific metadata accumulation characteristics that determine whether a data lakehouse maintains query speed over time.

This guide breaks down the core structural layers of the Iceberg format, walks through the exact mechanics of read and write paths, compares version capabilities, and details the maintenance disciplines required to keep production tables performant.

Key takeaways

• Iceberg defines tables through explicit metadata trees rather than folder layouts, freeing execution engines from directory scanning overhead.
• Table updates rely on an atomic catalog pointer swap using optimistic concurrency, ensuring serializable isolation and stable reader snapshots.
• Manifest files carry column statistics for every data file, enabling query engines to prune irrelevant files before reading storage.
• Every write appends new metadata files and data objects, making snapshot expiration and compaction essential operational practices.

What Apache Iceberg is and what it isn’t

Apache Iceberg is an open table format specification that brings database-like reliability and performance to large-scale data lakes. Originally developed by Netflix to overcome the scaling bottlenecks of legacy Hive tables, Iceberg is now an Apache Software Foundation project maintained by a broad open-source community.

To understand Iceberg, it helps to distinguish what it is from what it isn’t:

• It is an open specification, not a database or execution engine. Iceberg defines how table state, schemas, and file manifests are structured in metadata, allowing engines like Trino, Spark, and Flink to operate concurrently on the exact same tables.
• It is a metadata abstraction layer, not a physical storage engine. Iceberg sits directly on top of existing object storage, including Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS), MinIO, or HDFS.
• It tracks files explicitly, not by directory. Legacy formats like Apache Hive tracked tables by folder paths, requiring engines to list entire directories to infer which files existed. Iceberg eliminates directory listing overhead by maintaining an explicit, canonical inventory of every data file belonging to a table version.

The 3 layers of Iceberg

An Iceberg table operates across three distinct structural tiers: the catalog, the metadata layer, and the data layer.

The catalog

The catalog serves as the central point of truth for an Iceberg table, storing a reference pointer to the table’s current metadata file. Every commit is an atomic swap of this pointer. Either the new metadata file becomes current, or the commit fails and the existing pointer remains unchanged.

Because the catalog provides this atomic swap guarantee, choice of catalog directly impacts system interoperability. Options include the Iceberg REST catalog, Hive metastore, or managed cloud metastores. Starburst supports the Iceberg REST API spec across managed and enterprise deployments, providing a standardized catalog layer across diverse query engines.

The metadata layer

The metadata layer sits directly below the catalog and consists of three nested file types:

• Table Metadata Files: Written in JSON, these files record the table schema, partition specifications, current snapshot ID, and historical snapshot log. Every insert, update, or schema evolution appends a new table metadata file.
• Manifest Lists: An Avro file representing a specific table snapshot. The manifest list points to one or more manifest files and stores partition boundary summaries for those manifests.
• Manifest Files: Avro files that track individual data files along with column-level metrics, null counts, lower and upper bounds, and partition values. Query engines evaluate these statistics to skip entire data files during filter evaluation without scanning object storage.

The metadata layer also handles hidden partitioning. Instead of requiring users to supply explicit partition columns and directory paths in SQL queries, partition transforms sit in metadata. Users query base columns directly, and platform teams can evolve partition strategies over time without rewriting historical data files.

The metadata file is JSON. It holds the schema, the partition spec, the list of snapshots, and the current snapshot. Each snapshot points to one manifest list, and that list points to manifests. Snapshots can share manifests, so expiring a snapshot has to check what is still referenced. Manifest files list data files along with their partition values and column-level stats. Engines use those stats to skip files that can’t match a filter. Metadata does the pruning that directory layouts used to do. When you create a table, Iceberg writes a metadata folder. It holds the table’s first metadata file, which describes an empty table. Each insert, update, or delete then adds new manifests, a new manifest list, and a new metadata file. Those transactions and metadata changes are what each statement writes. Hidden partitioning also lives here. Partition transforms sit in metadata, so you query regular columns and never reference partition paths. Further, you can change a partition spec without rewriting old data.

The data layer

The data layer consists of the underlying physical storage objects holding your records. Iceberg stores data in optimized columnar formats, primarily Parquet or ORC.

Row-level updates and deletes write supplementary delete files alongside existing data objects rather than modifying files in place. Iceberg 2 supports position and equality delete files, while Iceberg 3 introduces binary deletion vectors to optimize deletion tracking. Data files remain immutable until an explicit maintenance job expires snapshots and purges unreferenced files from storage.

How reads and writes move through the layers

Iceberg guarantees ACID transactions on object storage by pairing immutable files with an atomic pointer swap in the catalog. Because writes never modify existing data in place, readers and writers operate concurrently without blocking one another.

The read path

When a query engine executes a SELECT statement against an Iceberg table, execution flows top-down through the metadata tree:

1. Catalog Resolution: The query engine asks the catalog for the table’s current metadata file location.
2. Snapshot Selection: The engine reads the table metadata JSON file and identifies the active snapshot ID (or a historical snapshot ID if performing time travel).
3. Manifest List Pruning: The engine opens the snapshot’s manifest list and evaluates partition boundary summaries to ignore manifest files that do not contain relevant records.
4. Manifest Pruning and Data File Selection: The engine reads the remaining manifest files, inspecting column min/max statistics and null counts to filter out specific data files.
5. Physical Storage Scan: The engine reads only the surviving Parquet or ORC data files from object storage and executes the remaining query logic.

The write path and commit mechanism

When an engine executes an INSERT, UPDATE, DELETE, or MERGE operation, execution flows bottom-up:

1. Data File Ingestion: The engine writes new, immutable Parquet or ORC data files (and any necessary delete files) to object storage.
2. Manifest and Manifest List Creation: The engine writes new manifest files tracking those data objects, followed by a new manifest list file representing the proposed table snapshot.
3. Table Metadata File Creation: The engine writes a new table metadata JSON file containing the proposed snapshot ID, updated schema, or altered partition specification.
4. Atomic Catalog Swap: The engine asks the catalog to atomically update its reference pointer to the new table metadata file.

Optimistic Concurrency Control

The catalog swap uses optimistic concurrency control (OCC). The update succeeds only if the catalog pointer has not changed since the writer began its operation.

If a competing writer commits first, the catalog rejects the second writer’s swap. The engine then inspects the new state, checks whether the concurrent write conflicts with its own changes, and either retries the commit against the updated snapshot or throws a serializable isolation failure.

Because readers lock onto a specific snapshot at query initialization, they are completely isolated from uncommitted writes, failed transactions, and concurrent updates.

How Snapshots, Branches, and Tags Work

Iceberg treats table state as a series of immutable historical points. By decoupling physical files from logical references, platform teams can run time travel queries, isolated staging branches, and audit freezes without copying underlying data.

Snapshots

Every committed write generates an immutable snapshot representing the complete state of the table at that specific microsecond. Snapshots enable time travel, allowing users to query historical table states for auditing or backfilling, as well as instant rollbacks to restore a known-good state if an ingestion pipeline writes corrupted data.

Branches

Branches act as mutable named references pointing to a specific snapshot lineage. They allow teams to create isolated workspaces within a single production table.

For example, data engineers can branch a main production table, write experimental transformations or run validation tests against that branch, and verify results without impacting production readers. Once quality checks pass, the branch can be merged back into the main branch through a fast-forward commit or select operation. Engine support for branch creation, writes, and merges varies, so verify query engine compatibility before building branch-based orchestration workflows.

Tags

Tags are immutable named references pointing to a fixed snapshot ID. Unlike branches, tags never advance or accept new writes.

Tags are ideal for marking quarterly financial closes, compliance baselines, or reproducible training datasets for machine learning. Assigning an independent retention policy to a tag protects its associated snapshot from automated snapshot expiration procedures, ensuring compliance data remains accessible even after standard table history is purged.

What changed from v1 to v3

The Iceberg format specification has evolved across three major versions, each addressing specific scale and performance bottlenecks in the data lakehouse.

Iceberg v1 – Analytic Data Tables

Version 1 decoupled logical tables from physical directory structures. By replacing implicit folder layouts with explicit metadata lists, v1 established file-level tracking, schema evolution, and basic snapshot isolation for append-heavy analytic workloads.

Iceberg v2 – Row-Level Changes

Version 2 introduced position and equality delete files to support row-level updates and deletes without rewriting full data files. While this enabled efficient upserts, it introduced read-side merge overhead. As delete files accumulate, query engines must reconcile base data files with multiple small delete files at scan time, degrading query performance until background compaction runs.

Iceberg v3 – High-Performance Lakehouse

Version 3 optimizes row-level operations and adds semi-structured data capabilities:

• Deletion Vectors Binary deletion vectors replace position delete files (which are deprecated in v3), significantly reducing read-side merge overhead during scans. Equality deletes remain supported.
• VARIANT Type Native semi-structured type support allows flexible querying of JSON-like data without full shredding.
• Extended Types and Features Introduces nanosecond timestamp precision, default column values, and native row lineage tracking across table commits.

Engine compatibility and migration risk should dictate your upgrade timeline. Because an Iceberg table cannot be read by an engine that does not support its format version, every query engine, ingestion job, and catalog service touching a table must fully support v3 before you upgrade the table format.

Why Metadata Design Requires Automated Maintenance

The exact architectural features that make Iceberg safe and performant, including immutable writes, optimistic concurrency, and granular file tracking, also cause metadata and physical storage footprint to expand over time.

Every write transaction appends new manifests, manifest lists, and snapshot records. High frequency streaming pipelines or frequent micro batch inserts generate thousands of small data objects and metadata files. Left unmanaged, this accumulation degrades platform performance in several ways:

• Small File Fragmentation: Thousands of tiny files force query engines to make excessive object storage API calls during scans, slowing down query execution.
• Metadata Bloat: Oversized metadata trees increase query planning overhead and raise commit latency for concurrent writers.
• Orphan File Storage Costs: Interrupted write operations, failed commits, or uncollected garbage routines leave unreferenced data files sitting in storage, driving up storage bills.

The Four Core Maintenance Tasks

Maintaining optimal query speeds and predictable storage costs requires four ongoing maintenance routines:

1. Data File Compaction: Merging small data files into larger, uniform blocks (typically 128 MB to 512 MB) and rewriting position delete files to eliminate read side merge friction.
2. Snapshot Expiration: Removing historical snapshot references past your compliance or audit window to keep metadata tree sizes manageable.
3. Orphan File Removal: Identifying and deleting unreferenced data and metadata files that were abandoned by aborted or failed job writes.
4. Manifest Optimization: Rewriting and sorting manifest files to align manifest boundaries with common query filtering patterns, sharpening file pruning efficiency.

Executing these maintenance jobs manually across hundreds of production tables becomes unmanageable at scale. Modern lakehouse platforms automate table maintenance routines in the background, executing compaction, snapshot expiration, and orphan file removal on dedicated background engines so query clusters remain focused on analytical workloads.

Scaling Apache Iceberg with Starburst

Apache Iceberg provides the open table format specification, but you still need a data platform capable of using it. Starburst provides the enterprise platform needed to operationalize and query it at scale. By pairing the open format with high-performance query execution, Starburst establishes the foundation of the modern open data lakehouse, enabling organizations to run fast, interactive SQL analytics directly over object storage without vendor lock-in.

Starburst enhances this open foundation with enterprise-grade security, automated table management, and cross-source query execution. Rather than locking data into proprietary warehouses or forcing massive migration projects, Starburst allows platform teams to query Iceberg tables alongside operational databases and legacy systems in a single SQL statement. Whether deployed as a fully managed cloud service or self-managed across hybrid and on-premises infrastructure, Starburst eliminates data duplication while delivering high-concurrency performance.

Accelerate Your Lakehouse Architecture

Ready to move from basic Iceberg tables to a high-performance, automated data lakehouse? Download The Data Engineer’s Guide to the Iceberg Data Lakehouse to learn best practices for catalog design, automated maintenance strategies, and multi-engine query optimization.

FAQs

What are the primary structural layers of Apache Iceberg? 

Apache Iceberg operates across three distinct structural tiers:

• The Catalog: Stores a pointer to the current table metadata file and executes atomic commits.
• The Metadata Layer: Consists of JSON metadata files, Avro manifest lists, and Avro manifest files that describe table schema, snapshot history, and file level statistics.
• The Data Layer: Contains the physical columnar storage files, primarily Parquet or ORC, along with any active delete files.

How does Apache Iceberg guarantee ACID transactions on object storage? 

Every write transaction generates new immutable data files and metadata records, culminating in an atomic swap of the catalog reference pointer using optimistic concurrency control. Because readers access a specific snapshot ID locked at query initialization, they are completely isolated from uncommitted writes, concurrent updates, and failed transactions.

How does Apache Iceberg differ from legacy Hive table designs? 

Apache Hive tracked tables implicitly through physical directory folder structures, requiring query engines to perform expensive directory listings to discover data files. Iceberg defines tables through explicit metadata trees, tracking every valid data file directly to eliminate directory scanning overhead and enable safe concurrent writes.

What is a snapshot, and how do branches and tags differ? 

A snapshot is an immutable point in time record of a table state after a commit. Branches are mutable named references pointing to a snapshot lineage, allowing teams to isolate staging and testing workflows before merging changes back into production. Tags are immutable named references to a specific snapshot ID, designed for audit baselines and reproducible analytics.

What changed in Iceberg v3, and when should teams upgrade? 

Iceberg v3 introduces binary deletion vectors to replace position delete files, native VARIANT data types for semi structured JSON data, nanosecond timestamp precision, default column values, and row lineage. Upgrade timing should be driven by engine compatibility, as every query engine and ingestion tool touching a table must fully support v3 before a table is upgraded.

Why do Apache Iceberg tables require ongoing maintenance? 

Because Iceberg relies on immutable writes and snapshot isolation, high frequency transactions accumulate small data files, bloated metadata trees, and unreferenced orphan files over time. Routine maintenance tasks, including file compaction, snapshot expiration, orphan file removal, and manifest optimization, are essential to prevent query planning slowdowns and rising storage costs.