Table maintenance is the unsung hero of the Iceberg world, but that doesn’t mean that it always gets the consideration it needs. Often, maintenance is configured once and never revisited, even if the table has long since outgrown that initial setup. In the worst case, it’s never configured at all. Either way, the table quietly degrades until the problem is too big to ignore. If this sounds familiar, you’re not alone.
This problem impacts data teams large and small. Queries that once finished in seconds start taking minutes. Meanwhile, storage costs creep up in ways that are hard to explain. Dashboards that were once reliable begin to lag. By the time the problem is obvious, it has usually been building for weeks.
Luckily, this problem can be solved because Iceberg tables degrade in predictable ways, and that helps us model and understand the degradation. For example, metadata often bloats as operations accumulate. In other cases, slowing query planning can increase commit latency. Other times, small files multiply faster than they can be efficiently scanned, dragging down query execution. Meanwhile, orphaned files from failed operations and expired snapshots quietly consume storage long after they have served their purpose. Knowing all of this helps us diagnose the problem and deploy a solution.
Starburst Icehouse LakeOps is that solution. Each of these issues has a direct fix, and each fix maps to a specific LakeOps capability. Let’s explore 3 ways that Iceberg can go wrong and how LakeOps can help automatically address each of these issues in the background without user intervention.
Problem #1: Bloated Metadata
The first place degradation shows up is in the metadata. As operations accumulate, the overhead of tracking them starts to outweigh the work itself. When left unaddressed, bloated metadata leads to worker memory pressure, slow query planning, and worsening commit latency, issues that LakeOps is designed to prevent.
Every INSERT, UPDATE, or DELETE operation on an Iceberg table produces one or more new manifest files that record the data files and delete files being committed. Over time, and especially with high-frequency workloads like singleton inserts, the number of these manifest files grows beyond what query planning can handle efficiently. The result is planning memory pressure, slower planning times, and increased commit latency. LakeOps addresses this by periodically rewriting manifest files, consolidating many smaller manifests into larger combined ones before the problem compounds.
Snapshots are a second source of bloat. Iceberg creates a new snapshot for each committed operation to support time travel and rollback, which means older snapshots accumulate and prevent storage from being reclaimed. LakeOps handles this through periodic snapshot expiration, freeing up storage from data and metadata files that are no longer referenced by the current snapshot.
A less obvious source of bloat is dangling deletes. Normally, delete files are removed from the iceberg metadata when they are no longer needed, but in some cases this automatic cleanup is not possible. As a result, millions of delete files may be kept in the metadata and processed, even though they’ll never apply to any data files. LakeOps identifies and removes these dangling deletes, keeping the metadata lean regardless of how the table is partitioned.
Problem #2: Slow Query Execution
The second place degradation shows up is in the query execution itself. Metadata bloat slows planning, but how data is physically organized across files determines how fast queries actually run.
The most common culprit is small files. Frequent singleton inserts can produce hundreds of Parquet files, each holding only a single record. The compute engine (in this case Trino) has to download, open, and read every one of them, work that could be done far more efficiently against a handful of larger, consolidated files. LakeOps’ data compaction process addresses this directly, merging small files into optimally sized ones so that scans cover more data with less overhead.
Delete files introduces a separate class of performance problems. In Iceberg V2, position delete files identify deleted rows by filename and position, but a single position delete file can reference deletions across many data files. Trino may end up reading a very large delete file just to process a single deleted row from the file it is actually scanning. Iceberg V3 mitigates this by replacing position delete files with deletion vectors, but both implementations become inefficient when most rows in a scanned file have been deleted. Trino is forced to process thousands of deleted rows to return only a handful of live ones. Equality deletes carry their own cost. Each one requires an expensive row filter operation applied to every row in every data file.
Compaction solves both problems. When LakeOps compacts a table, it does not just consolidate files. It also writes the newly generated data files without including the deleted records, eliminating the overhead of reading and resolving those deletes at query time entirely.
Problem #3: Wasted Storage
The third problem is quieter than the first two but just as costly over time. Storage fills up with files that are no longer needed but have not been cleaned up.
Some of this is by design. When LakeOps compacts data files or rewrites manifests, it generates new, more efficient snapshots. But the original files from previous snapshots remain accessible to support point-in-time queries, and they stay that way until those snapshots expire. Once they do, the files they referenced become orphaned, sitting in storage with nothing pointing to them.
Failed operations create a separate category of waste. When an operation fails mid-execution, it can leave files behind that were never committed to the table metadata. These have no snapshot to expire, no cleanup trigger, and no natural path out. They simply accumulate.
LakeOps orphan file removal handles both cases. It scans the table metadata, identifies files that are no longer referenced by any snapshot, and deletes them, reclaiming storage that would otherwise grow unchecked in the background.
Leverage Starburst LakeOps today
Left unattended, Iceberg tables degrade in ways that are predictable, compounding, and entirely avoidable. Metadata bloats until query planning slows to a crawl. Small files accumulate until scans that should take seconds start taking minutes. Storage fills with orphaned files that serve no purpose but cost real money. None of these happen all at once, which is exactly why they tend to go unnoticed until the damage is done.
The good news is that each failure mode has a known fix. Manifest rewrites and snapshot expiration keep metadata lean. Compaction consolidates files and eliminates the overhead of reading deletes at query time. Orphan file removal reclaims storage that would otherwise grow quietly in the background. What Icehouse LakeOps does is take all of that work off your plate entirely, running it automatically, continuously, and without requiring anyone to remember to do it. It can even handle extremely large unhealthy tables that other engines find difficult.
Seen this way, LakeOps really is designed to solve three core problems:
- LakeOps makes table maintenance easy to manage and deployed in a serverless manner, which means there is no need to size and manage clusters, and no need to schedule work.
- LakeOps can even handle extremely large, unhealthy tables that other engines find difficult.
- LakeOps is designed to compact incrementally throughout the day, meaning that there are no long delays until maintenance starts.
Overall, table maintenance matters more than it might seem. Doing table maintenance correctly at scale is not straightforward. Done wrong, it can mean corrupted tables at best and silently wrong query results at worst. Done right, it means a lakehouse that stays fast, stays lean, and stays trustworthy as it grows.
Ready to know more about LakeOps it in action? Check out our LakeOps webinar, or read more about the feature on our Starburst Documentation for Icehouse LakeOps.
