Where table formats stop
By Dylan Wolpe
The short version
- Iceberg and Delta solve table management, atomic commits, schema evolution, snapshot isolation, and they solve it far better than what came before.
- Time travel is a convenience feature backed by retained files, not a tamper-evidence mechanism: it reads metadata that the same privileges which could alter the data can also alter.
- Neither format offers per-subject erasure. Deleting one person means rewriting the affected files, which produces new files and a new snapshot rather than a provable erasure.
- They are complementary to an archive layer, not competing with one, the sensible arrangement is table formats for the working set and a verified format for the cold, regulated tail.
Table formats fixed a genuinely painful problem. Before them, a data lake was a directory of files and a set of conventions everybody hoped were being followed. Iceberg and Delta brought atomic commits, schema evolution and snapshot isolation, and the improvement was enormous.
They were designed for tables under active use. The trouble starts when their features are read as answers to questions they were never asked.
What they genuinely give you
- Atomic commits, so a reader never sees a half-written update.
- Schema evolution without rewriting history.
- Snapshot isolation for concurrent readers and writers.
- Time travel, querying the table as it stood at an earlier point.
- Partition and file pruning that makes large tables tractable.
If you manage an analytics estate and are not using one of them, that is the more urgent decision. Nothing here is an argument against them.
Where the mapping to compliance breaks
Time travel is not an audit trail. It works by retaining old data files and the metadata pointing at them, and it is read through the catalog. Someone with sufficient privileges to modify the table generally also has the privileges to expire snapshots and rewrite metadata. Time travel is superb for recovering from a bad job at 3am. It is not designed to withstand someone who wants the history to have been different, and those are different threat models.
There is no per-subject erasure.Deleting one person means rewriting every file containing their rows. That produces new files and a new snapshot, and the old files remain until snapshot expiry, which is a separate operation people routinely forget, leaving the “deleted” data recoverable through time travel by design.
| Requirement | Table format | Archive layer |
|---|---|---|
| Atomic writes, schema evolution | Yes | Not its job |
| Concurrent readers and writers | Yes | Not its job |
| Recover from a bad job | Yes | Partially |
| Prove nothing was altered | No | Yes |
| Erase one subject provably | No | Yes |
| Readable in fifteen years | Probably | Designed for it |
Time travel answers “what did this look like on Tuesday?”. An auditor asks “prove nobody changed Tuesday.” Those are not the same question.
The arrangement that works
Split by temperature and by requirement rather than choosing a winner.
Working set in a table format. Data being queried, appended and evolved. Concurrency and schema flexibility are exactly what you need, and integrity requirements are typically satisfied by ordinary access control.
Cold, regulated tail in an archive format. Data past its active life but inside a retention obligation. Nothing is being written. What matters is that it remains provably unaltered, individually erasable, readable in a decade, and searchable without a full restore.
These are genuinely different jobs, and the reason one tool does both badly is that the second job’s requirements, immutability, provable erasure, no dependence on a catalog, are close to the opposite of the first’s.
The question that finds the boundary
Which of your tables have not been written to in twelve months but cannot be deleted?
That set is almost always larger than people expect, it is paying working-set prices for cold-set access patterns, and it is the part an auditor will eventually ask about. It is also where the storage difference compounds, because it never gets smaller on its own.
Questions people ask about this
Can Iceberg or Delta time travel serve as an audit trail?
Not as evidence. Time travel works by retaining old data files and metadata, and it is read through the same catalog that a privileged user can modify. It is excellent for recovering from a bad job or comparing yesterday's numbers. It was not designed to be defensible against someone with write access who wants history to look different.
How do you delete one person from an Iceberg table?
By rewriting every file containing their rows, which creates new files and a new snapshot. It works, it is expensive at scale, and it produces no artefact proving that only that person's data changed. Expiring the old snapshots is then a separate step people frequently forget, leaving the data recoverable.
Should we use a table format or an archive format?
Both, for different data. Table formats are built for data that is actively queried and evolving. An archive format is for the cold, regulated tail where the requirements are integrity, provable erasure and long-term readability rather than concurrent writes.