Not So Fast: Equality Deletes, Compaction, and Iceberg V4

Equality deletes are expensive to read, easy to accumulate, and awkward for row lineage and change extraction. Alex Merced’s Why Iceberg V4 Wants to Retire Equality Deletes lays out those costs and describes an alternative for streaming writers as keep producing equality deletes, then convert them to deletion vectors (DVs) in the background.

We agree with much of that argument. On the Icehouse team, we built LakeOps, which runs automatic maintenance for Iceberg tables, including compaction, delete application, and snapshot expiration. Equality deletes have been one of the hardest parts of that work.

But they also have a useful property that is easy to overlook. A delete expressed by value can survive a change in where the row is stored. That lets streaming updates and compaction proceed independently in a way that file-position-based deletes cannot.

The linked post mentions commit conflicts with other maintenance. For high-update streaming tables, that deserves more than a passing mention. Compaction has to make progress while updates keep arriving.

A version note before we get into the mechanics. As of October 22, 2026, Iceberg’s upstream V4 draft prohibits writing new equality deletes, while retaining support for reading existing ones. The specification still marks V4 as under development and not formally adopted. The concurrency behavior discussed below is from the V3 implementation, checked against Iceberg 1.12.0 and the upstream revision linked at the end.

Why streaming writers use equality deletes

Iceberg V2 introduced two ways to describe a row-level delete:

• Position deletes identify a row by location: “row 4,812 in data-00042.parquet is deleted.” V3 uses deletion vectors for new position-based deletes, encoding those positions as a bitmap.
• Equality deletes identify rows by column values: “delete rows where order_id = 9137.” Their scope also depends on partition and sequence-number rules.

A CDC upsert can write an equality delete for a key and insert the new version of the row in the same commit. Equality deletes apply to data with a strictly lower data sequence number, so the delete removes older versions without deleting the replacement row from that commit.

That gives the streaming writer two advantages.

The writer does not need to locate the old row

To write a position delete or DV for an existing row, the writer needs its file path and row position. That means scanning data or maintaining an index.

An equality delete only needs the key. For a sink committing every 30 seconds, avoiding a read before each write is a substantial simplification.

The work does not disappear. Readers and maintenance jobs eventually have to match those keys against data. Equality deletes move that work away from the ingestion path.

The delete remains meaningful after compaction

Suppose compaction starts at snapshot sequence number 100. It reads the selected files, applies the deletes visible in that snapshot, and writes replacement files.

With Spark’s default use-starting-sequence-number=true, those replacement files receive the starting snapshot’s sequence number, 100, rather than the sequence number of the eventual compaction commit. This is not the same as preserving each input file’s original sequence number. LakeOps compaction follows the same approach, assigning replacement files the sequence number of the snapshot at which compaction started.

Now suppose ingestion commits an equality delete at sequence number 101 while compaction is still running. That delete can apply to the replacement files because their data sequence number is still 100. The row may have moved, but the key-based delete still describes it.

Iceberg’s rewrite validation accounts for this. A rewrite using the starting sequence number does not reject concurrent equality deletes merely because they might match the rewritten data. Other conflicts can still fail the commit, but a new equality delete does not, by itself, invalidate the rewrite.

That is the concurrency property we want to preserve.

Streaming tables need compaction

Low-latency ingestion tends to produce small files. A sink committing every minute can write files for each active partition and writer task, plus delete files. Over a day, a busy partition can accumulate thousands of files.

Without maintenance:

• More files and manifests increase planning work.
• Scans pay file-opening and other per-file overhead repeatedly.
• Merge-on-read deletes accumulate, making every query do more work.

Compaction is therefore part of operating a streaming table, not an occasional cleanup. A design for streaming updates has to account for ingestion and compaction overlapping.

Equality deletes help with that overlap. They also make the compaction itself harder.

Making equality-delete compaction recover from a backlog

To rewrite data files correctly, compaction must apply every equality delete that could match their rows. For each match, it must respect the row’s data sequence number and the delete’s sequence number.

The reference Java reader loads applicable equality-delete keys into in-memory sets. When CDC keys are spread across a partition, file-level bounds may do little to prune the deletes. Many delete files can remain applicable to many data files.

That works until the delete-key sets outgrow memory.

A backfill, an update burst, or a maintenance pause can create a backlog large enough to make compaction run out of memory. Meanwhile, ingestion keeps adding deletes. The next attempt has more work than the one that failed.

In LakeOps, we built a batched delete-application path to break that loop:

1. Order equality-delete files by sequence number and divide them into batches sized for a memory budget.
2. Apply one batch at a time. Load its keys, stream the data through, and remove rows matched by a newer delete.
3. Write surviving rows to local disk. They become the input to the next pass.
4. Repeat until the final batch, then write the compacted output.

The important detail is preserving the sequence-number semantics across passes. Intermediate rows cannot simply be treated as newly inserted data. Each later batch has to make the same decision it would have made against the original rows.

We trade extra local I/O for bounded delete-key memory. In our workloads, the batch count usually stays small, and multiple passes are needed only when the backlog is large. Compaction can then recover from backlogs that previously caused repeated out-of-memory failures.

This is a fair point in favor of retiring equality deletes: supporting them well takes engineering work. Our batching approach is one way to handle that cost. It does not remove the cost, and every engine need not use the same approach.

The harder question is what replaces their concurrency behavior.

What background DV conversion solves

Flink’s experimental ConvertEqualityDeletes task, available in Iceberg 1.12.0, moves row-location lookup out of the sink’s direct write path. In the separate-branch setup:

1. The sink writes data and equality deletes to a staging branch.
2. The converter maintains a checkpointed primary-key index in Flink state and resolves keys to file paths and row positions. RocksDB is a recommended state backend, not a requirement of the conversion algorithm.
3. The converter publishes the corresponding data and DVs to the target branch used by readers.

Readers get position-based deletion, and ingestion can keep writing key-based changes. The target branch’s freshness depends on conversion keeping up.

That is a useful design. But the target branch now contains deletes tied to particular data files, so it still needs a way to coexist with compaction.

How DVs and compaction collide

Suppose compaction starts rewriting files A, B, and C into X. It reads the input files, applies known deletes, and writes the output. On a large group, that can take minutes.

While it runs, the converter commits a new DV for B.

Compaction:  read A, B, C ── apply known deletes ── write X ── commit fails
Converter:                  commit a new DV for B

The compaction output does not include that newly committed delete. Nor can Iceberg attach B‘s DV to X: its positions refer to B, and the corresponding rows in X may have different positions.

The V3 rewrite validation rejects the commit when it finds a new position-based delete against a file being replaced. A metadata commit retry cannot make the output correct. The rewrite must be redone or repaired through a mechanism that accounts for the new delete.

If compaction commits first, the converter has the opposite problem. It planned deletes against files that are no longer live on the target branch. It must resolve the keys against the new layout rather than publish stale positions.

Either ordering can invalidate work. When updates are frequent and spread across many files, a long-running rewrite can repeatedly encounter new deletes before it commits. If successful rewrites cannot keep pace with ingestion, small files and delete work keep accumulating.

This is a workload-dependent failure mode, not a claim that every DV-based streaming table will stall. The dangerous combination is a long rewrite window and frequent updates to the files inside it.

Who merges concurrent DVs?

V3 allows at most one DV per data file in a snapshot. A writer adding deletes to a file must include its existing position-based deletes in the replacement DV.

That raises an important implementation question: if another writer commits a newer DV while ours is being prepared, does Iceberg merge them automatically?

The V3 commit path does not automatically merge a DV introduced by a concurrent snapshot into an already prepared DV. There are several distinct responsibilities:

• Building a DV: the library’s BaseDVFileWriter can combine new positions with previously loaded deletes for that data file.
• Combining inputs to one commit: MergingSnapshotProducer.mergeDVs can merge multiple DV inputs submitted within the same snapshot update. This is not a rebase onto another writer’s newer snapshot.
• Checking concurrent commits: the commit validator rejects a newly added DV for the same referenced data file. A catalog metadata retry does not recompute the prepared bitmap against that newer DV.
• Recovering the operation: the writer or orchestration layer must refresh its view and rebuild the affected work. If data files were rewritten, that also means resolving row locations again.

Flink’s equality-delete converter handles this through its own recovery path. When it has positions to write and both target snapshot IDs are available, it checks that the current target snapshot still matches the one used for planning. Any snapshot-ID change, even from an append-only commit, fails that pre-write check. The commit stage separately validates relevant file changes from the planned target snapshot, covering races that arrive after the check.

On a subsequent trigger, the planner detects the changed target, clears and rebuilds the target index, and recomputes the conversion. It does not ask the core commit library to merge the stale DV into the latest one.

The direct Flink sink follows a different path: its upsert writer produces equality deletes. Its committer assumes any position deletes concern data files added in the same commit; it is not a general DV updater for existing rows. ConvertEqualityDeletes is the maintenance component responsible for producing DVs and replanning after target-branch conflicts.

So the answer is split: the library provides DV construction, merging helpers, and correctness validation; the converter owns replanning and recomputation after a concurrent target change. The existing helpers do not remove the contention window.

What coordination can—and cannot—do

There are practical ways to reduce conflicts. Their effectiveness depends on the workload and on who controls the writers.

Staggering schedules helps when conversion and compaction can fit into separate windows. A converter running every five minutes and a rewrite taking fifteen minutes do not naturally have such a window.

Shared locking can serialize the operations. Flink’s maintenance framework supports JDBC- and ZooKeeper-backed locks that can coordinate maintenance across jobs. This is not limited to one Flink job, but every relevant participant must use the same locking protocol and table identifier. An external Spark job or maintenance service does not automatically participate.

A platform controlling both conversion and compaction can make that coordination practical. The trade-off remains: conversion waits while compaction holds the lock. In a separate-branch setup, staging ingestion may continue, but target-branch visibility falls further behind during the wait.

Smaller rewrite groups and partial progress shorten each commit window and preserve successful work. They can substantially improve progress. They do not guarantee it when updates repeatedly reach the files in each group.

These techniques are useful operating tools. Equality deletes avoid this particular coordination requirement because their meaning survives a file rewrite.

What we see in customer workloads

Two LakeOps workloads illustrate why the distinction matters.

One customer runs CDC ingestion into tables with thousands of partitions. Each ingest commit updates rows across a large fraction of those partitions. With that overlap, position-based deletes were not a realistic choice for their maintenance model. Equality deletes let ingestion and compaction run side by side, and our batched delete application helps compaction stay ahead of the backlog.

Another customer has a much lighter workload: a periodic delete job using position deletes, at a fraction of the first customer’s update rate. Even there, conflicts with compaction occur often enough that the delete job needs retry handling to finish reliably.

These are different kinds of evidence. The first illustrates a workload that depends on key-based deletion; the second is an observed position-delete conflict. Neither is a benchmark of Flink’s DV converter. Together, they explain why we would want to test sustained concurrent progress before replacing equality deletes in a high-update CDC design.

What V4 needs to preserve

DVs have compelling advantages. They give readers a compact, direct way to determine which positions are deleted, and they avoid repeated key matching. Equality deletes also complicate row lineage: an upsert that does not read the old row cannot carry its row ID into the replacement. Identifying deleted rows for change extraction may require scanning data to match keys.

But equality deletes serve two purposes:

1. They let a streaming writer express an update without first locating the old row.
2. They let that delete remain valid when compaction moves the row to another file.

Background conversion addresses the first purpose. The V3 conversion and commit paths examined here still need coordination or recomputation to handle the second.

One possible direction is compaction that repairs its output after a delete conflict. A rewrite could retain a mapping from input row positions to output positions. If a new DV arrives, it could translate those additional deletes into a DV for the output rather than reread and rewrite all the data.

That mapping is not free, especially for sorting and other rewrites that reorder rows. Committing the fix-up would also need to handle further concurrent changes safely. This is a design direction, not a capability provided by the current V3 rewrite path, and it would need support beyond a scheduler.

The requirement is broader than any one implementation: a replacement for equality deletes should let ingestion and maintenance make sustained progress together, with acceptable freshness and retry costs.

We have spent real engineering effort making equality deletes manageable, so we understand the motivation to retire them. Before doing so, we should preserve the property that makes them valuable to high-update streaming tables: deletes that remain meaningful while compaction changes where the rows live.

Implementation references

The source review used Apache Iceberg 1.12.0 and upstream commit 60897e0b828580cc20e591424179d9b5c574e8d6. V4 draft wording is from the upstream commit; the release’s specification still describes V4 as under development.

• Specification: scan planning and delete applicability.
• Specification: delete formats and the one-DV-per-file rule.
• Spark rewrite options, including the starting sequence number and partial progress.
• Flink maintenance documentation: conversion, state, locking, and conflict recovery.
• Core commit validation rejects concurrently added same-file DVs, while mergeDVs combines inputs to one pending operation.
• Core snapshot commit retries metadata conflicts, not failed delete validation.
• Rewrite validation distinguishes equality deletes from position-based deletes.
• Flink sink commit assumptions for position deletes.
• Converter pre-write snapshot check, commit validations, and index rebuilding on a subsequent trigger.