Parquet — but with integrity and a delete button.
Parquet is a great columnar format, and AT-1 doesn't ask you to leave it — your engines still read Parquet, byte-identical. What AT-1 adds is the two things Parquet can't do: prove the file was never altered, and erase one person for POPIA or GDPR. And in cold-tier mode it stores the table ~27% smaller than Parquet+zstd along the way.
| Format | Queryable | Size | Tamper- evident | Per-subject erasable |
|---|---|---|---|---|
Parquet excellent columnar format — but no integrity proof and no way to erase a subject | good | |||
Parquet + zstd smaller, same two gaps | better | |||
AT-1 reads as Parquet, ~27% smaller in cold tier, tamper-evident + per-subject erasable | best* |
* Cold-tier columnar mode is ~27% smaller than Parquet+zstd on real NYC-TLC. In fully-regulated mode, per-subject encryption adds overhead and Parquet can edge us on raw ratio — the trade for a per-person delete.
The two gaps AT-1 closes
Prove it wasn't altered
A SHA-256 manifest binds the file. Change one byte and verification fails — so a Parquet table becomes provably the original, or provably not. Parquet has no such proof.
Erase one person
A right-to-erasure request destroys one subject's key in milliseconds; their retained rows stay queryable and the bytes never move. Parquet can't delete a record without rewriting the file.
Parquet-alternative questions, answered
- Do I have to replace Parquet in my pipeline?
- No. AT-1 reads back as Parquet — Spark, Trino, DuckDB, Snowflake and Databricks return byte-identical results and cannot tell the difference. Your catalog, table definitions and queries are unchanged. You're swapping the bytes at rest, not the format your engines see.
- Is AT-1 actually smaller than Parquet?
- In cold-tier (columnar) mode, yes: on real NYC-TLC, AT-1 stored a table about 27% smaller than Parquet+zstd-9, byte-for-byte lossless. In the fully-regulated mode — where each data subject gets their own encryption key so you can erase them — that per-subject overhead means Parquet can edge us on raw ratio. We're honest about which mode you're in: pick columnar for maximum size win, regulated when you also need per-subject erasure.
- What does AT-1 add that Parquet can't do?
- Two things Parquet fundamentally can't: prove the file wasn't altered (a SHA-256 manifest fails on a single changed byte), and erase one data subject for POPIA/GDPR without rewriting the archive. Parquet has neither integrity proof nor a per-record delete — AT-1 adds both while keeping Parquet's queryability.
- Why not just use Parquet and handle integrity/erasure separately?
- You can — with an external hash ledger and a bespoke deletion process — but that's two more systems to build, run and keep in sync, and the deletion still means restoring and rewriting the Parquet. AT-1 folds integrity and per-subject erasure into the artifact itself, so they travel with the data and don't cost you a full rewrite.
Keep Parquet. Add integrity and a delete button. Bring a sample and we'll prove it.
Get in touch