27.3% smaller than Parquet+zstd-9 on real taxi data — and four places it loses

Compression benchmarks are usually run on data chosen by the vendor, against a baseline configured by the vendor, reporting a number chosen by the vendor. So here is a specific one on public data, with the baseline tuned in the baseline’s favour, and the cases where we lose listed in the same article.

The setup

Official NYC Taxi & Limousine Commission yellow-taxi data: 1,000,000 rows, 19 columns. Pickup and dropoff timestamps, coordinates, passenger counts, distances, a set of fare components, and several low-cardinality categorical codes. It is a genuinely representative analytics table — not a synthetic best case, and not something we generated.

The baseline is Parquet with zstd at level 9, which is a real production configuration rather than a strawman. Comparing against uncompressed Parquet, or against zstd-1, would inflate our number by a wide margin and tell you nothing.

Where the 27.3% comes from

Not from a better general-purpose compressor. If you swap zstd for a stronger byte-oriented codec you get a few percent, not twenty-seven, because the compressor is already close to what is achievable when it treats the data as a stream of bytes.

The gain comes from not treating it as bytes. A fare amount is a decimal with two places, not a float and not a string; stored as a scaled integer it compresses very differently. A timestamp column is monotonic within a partition. A pickup coordinate is one of a bounded set of real locations. Trip distance and fare are correlated. Each of these is a modelling decision made per column, and the residual after modelling is what gets compressed.

The saving is proportional to how much of your table is structured. That is a real advantage on analytics data and close to worthless on a table of hashes.

Four places it loses

1. High-entropy columns. UUIDs, hashes, encrypted blobs, compressed payloads in a column. There is no structure to model, so we floor out at roughly what a general compressor achieves. On a table that is mostly identifiers, expect single-digit gains or none. We consider flooring honestly here a feature — a tool that claimed a big win on random data would be broken — but it is still a loss.

2. Natural photographs and video. We lose to WebP and to x265, and it is not close. Those codecs encode perceptual models built over decades. Where we do win in imaging is 16-bit medical, depth and astronomical data, where lossless matters and the general-purpose photo codecs are a poor fit. If someone is selling you one format that beats everything at both, check that claim first.

3. Tables that are already narrow and dense. If the schema is three integer columns with high cardinality, the modelling has little to find and the overhead of carrying the model starts to matter. The gain is roughly proportional to how much decimal, temporal and categorical structure exists; strip that out and the advantage strips out with it.

4. Pure archival storage cost, in isolation. This is the one worth being blunt about. If your only requirement is smallest bytes at rest and you never need to read the data again, a general-purpose compressor at maximum settings gets you close enough that the difference rarely justifies changing formats. We have measured this on our own database corpora and said so publicly: on storage alone, we land near xz. Storage is not the moat.

Why we think this is the less interesting number

A 27.3% storage reduction on a cold tier is worth real money at petabyte scale, and for some buyers that is the whole business case. But it is a quantitative advantage, and quantitative advantages get competed away. Someone will publish 30% next year.

The properties we think actually matter are the ones Parquet does not have at any setting. You can query an AT-1 archive in place without rehydrating it. You can verify that it has not been altered, and if it has, find out exactly where. You can erase one data subject from it without changing the file’s hash — which is the difference between an archive that is a compliance liability and one that is a compliance asset.

Those are categorical rather than incremental, and they are why the honest pitch is not “smaller Parquet”. Smaller is the part that happens to also be true.

Checking it

The dataset is public, the baseline configuration is stated, and the same rows go through every path. We publish the full benchmark set — including the domains where we find nothing — on the comparison page. If you reproduce this and get a materially different number, we would genuinely like to hear about it; that is a more valuable email than most of the ones we get.

• The lakehouse case in detail: cold tier, queried in place
• Why we run benchmarks this way: A ratio is not a benchmark
• Measure your own data: free compressibility audit