What does Apache Iceberg do ?

• it manages large slow-changing tabular data and gives a sql interface to the data so that it can be queried efficiently
• breaks files into partitions and stores those files into an object store such as s3. partitions can be filtered based on the partition key(s). the partitioning is “invisible partitioning” meaning it is done by the system for you, without exposing the details to the client.
• it separates out metadata management from the data. metadata is not stored in the data files.
• it separates table schema away from the data . a change of column name will not affect the data files. see Schema Evolutuon.
• it allows accessing data as it existed at a specific point in time. this Time Travel feature is useful for auditing, debugging and reproducing issues that occurred in the past . Time travel is implemented using “snapshot isolation” which allows multiple versions of the same table to exist at the same time. (Copy on Write is used in the implementation)
• it provides ACID compliant transactions for data modifications and snapshot isolation for queries, which help ensure consistency and correctness of data
• it does all this through a lightweight design with minimal coordination

Figure. iceberg table format is used by multiple engines and is capable of writing to multiple storage types. source.

Ryan Blue’s discussion on the rationale for the design is here and a presentation with performance improvements is at https://conferences.oreilly.com/strata/strata-ny-2018/cdn.oreillystatic.com/en/assets/1/event/278/Introducing%20Iceberg_%20Tables%20designed%20for%20object%20stores%20Presentation.pdf

“By building support for Iceberg, data warehouses can skip the query layer and share data directly. Iceberg was built on the assumption that there is no single query layer. Instead, many different processes all use the same underlying data and coordinate through the table format along with a very lightweight catalog. Iceberg enables direct data access needed by all of these use cases and, uniquely, does it without compromising the SQL behavior of data warehouses.”

The client is a java jar file which can be embedded.

How does iceberg store files in s3 ?

The top level directory contains the table’s metadata files including the schema and partition information. The metadata files are stored in S3 object store using the table name as the s3 prefix.

The data files are stored in a directory structure that reflects the table partitioning. Partition values are encoded in the directory name.

s3://bucket-name/table-name/date=YYYY-MM-DD/region=us-west-1/0001.parquet

s3://bucket-name/table-name/date=YYYY-MM-DD/region=us-west-1/0002.parquet

Why a new table format – https://github.com/Netflix/iceberg

A hands-on look at Iceberg table by dremio is here .

A blog on the Adobe experience with Iceberg is here.

A blog on creating a real-time datawarehouse with Flink and Iceberg – https://www.alibabacloud.com/blog/flink-%20-iceberg-how-to-construct-a-whole-scenario-real-time-data-warehouse_597824

What are the insights behind iceberg, that lead its rapid adoption ? What were the old assumptions about data warehousing that were being challenged ?

The big changes in data warehousing were mostly driven by four old assumptions breaking.

The first assumption was that storage and compute should be tightly coupled inside one database system. Classic warehouse systems were built around local or tightly managed storage attached to the query engine. As data volumes grew and cloud economics took over, this became too rigid: teams wanted cheap durable storage, elastic compute, and multiple clusters reading the same data. Dremel’s evolution into BigQuery explicitly highlights disaggregated storage and compute as a foundational design principle, and Snowflake’s core architecture is likewise a shared-data, multi-cluster design. Redshift’s later evolution also emphasizes tiered storage, auto-scaling, and cross-cluster sharing. This meant loose coupling between storage and compute.

The second assumption was that the database engine should own the data format and the only serious workload was SQL analytics. That held when warehouses were mainly fed by ETL and queried by BI tools. It weakened when the same data also needed to support data science, machine learning, streaming ingestion, and direct file access. The lakehouse literature describes this as the pressure to combine warehouse features with open direct-access formats such as Parquet and ORC, instead of forcing everything through a closed warehouse storage layer. So this meant loose coupling between the database backend and the application workload.

The third assumption was that catalog metadata was mostly about table definitions, not about exact evolving table state. In Hive-style systems, metadata was largely names, columns, partitions, locations, and storage descriptors. That worked until tables became huge, mutable, multi-engine, and compliance-sensitive. Then it was not enough to know what a table is called and where it lives; systems also needed to know exactly which files are live now, which rows are logically deleted, which schema version applies, and how to time-travel or roll back. Iceberg’s spec is a response to that pressure: it adds snapshots, manifest lists, manifests, delete files, partition specs, and hidden partitioning so that the table’s current state is metadata-defined rather than directory-defined. 

The fourth assumption was that append-oriented batch processing was enough. Once organizations wanted late-arriving data, updates, merges, GDPR deletions, streaming-to-analytics pipelines, and reproducible historical queries, append-only file piles became operationally awkward. Modern lakehouse systems therefore added row-level operations, snapshot isolation, and richer table metadata on top of object stores. Recent work on Iceberg-based row-level operations is explicitly about making those mutable operations practical at very large scale.

Here’s a metadata explorer to compare the metadata stored in iceberg and hive formats – metadata explorer app.