A compression ratio is not a benchmark

Every compression vendor, us included, has an incentive to publish the largest defensible number. The defensible range is wide — the same tool on the same data can honestly produce 1.2× or 40× depending on choices nobody is obliged to disclose. So the ratio is not the claim. The methodology is the claim, and the ratio is a footnote to it.

Here are the six questions we use on other people’s benchmarks, and that we have failed on our own more than once.

1. What is the baseline, and how hard was it trying?

The oldest trick, and mostly not deliberate. You tune your own encoder for weeks, then compare against whatever the baseline does out of the box. Against gzip at default settings, almost anything looks revolutionary. Against zstd at level 19, or xz at maximum, the honest gap is usually much smaller and sometimes reversed.

The rule we adopted after getting this wrong: the identical codec set on both sides. Whatever family of compressors is available to our method must be available to the baseline, at comparable effort. If we take the best of four compressors, so does the baseline. A win that survives that is a win; one that doesn’t was an artefact of letting our side try harder.

2. Lossless, or “lossless”?

This word does more unearned work than any other in the field. It can mean the reconstruction is bit-identical to the input, verifiable by hash. It can also mean visually indistinguishable, or statistically equivalent, or lossless-after-a-preprocessing-step that quietly discarded precision.

The test is one question: does the SHA-256 of the reconstruction equal the SHA-256 of the original? If yes, the claim is unambiguous and checkable by anyone in a minute. If the answer involves explanation, you are being offered a weaker guarantee under a stronger word.

3. What is the dataset, and who chose it?

Structure-aware compression is only as good as the structure present. On rule-governed data — logs, market ticks, genomics, telemetry, game records — modelling wins decisively. On high-entropy data it floors out at roughly what a general compressor achieves, and it should.

This makes dataset selection the single largest lever on the headline number, which is why the dataset has to be named, public where possible, and not curated to the method. Ours is stated on every result; where we use public corpora we say which, and where the data is a customer’s we say that too, and publish the shape rather than the bytes.

4. Is the artefact you are testing the artefact they ship?

We learned this one the embarrassing way. An external evaluator ran our published download and got materially worse numbers than our internal runs. Nothing had been misrepresented — the distributed binary was simply behind the code the results came from. From where they were standing, that is indistinguishable from inflated claims, and they were right to mark us down for it.

The lesson generalises past compression: a claim is only as mature as the artefact a stranger can download.We now gate releases on reproducing the published numbers from the shipped build in CI. If a vendor’s benchmark cannot be run against the thing you can actually install, it is a research result, not a product claim — both are legitimate, but only one of them is what you are buying.

A benchmark that only reproduces on the vendor’s machine is a description of the vendor’s machine.

5. Where is the control?

This matters most when compression is used as a measuring instrument rather than a storage tool — detecting structure, anomalies, drift. A compressor is exquisitely sensitive, and sensitive instruments find things.

We ran a randomness audit on a national lottery and initially found a dramatic bias. The “bias” was a change in the game’s format part-way through the history we had pooled. The detector was working perfectly; the experiment was badly designed. The fix is always a control that has been through the identical pipeline, so that artefacts of the pipeline cancel and only real effects survive. We wrote that one up in full, including the wrong answer: the PowerBall audit.

6. Where does it lose?

The most informative question you can ask any vendor, and the cheapest to ask. A method that wins everywhere either has not been tested widely or is being described selectively. Real methods have shapes: they are strong somewhere and weak elsewhere, and knowing the boundary is what lets you predict your own result.

Ours, stated plainly:

• Natural photographs and video — we lose to WebP and x265, clearly. We win on 16-bit medical, depth and astronomical imaging, where lossless is required.
• High-entropy data — encrypted, random, already-compressed. We floor out. Any tool claiming a large win here is broken or measuring something else.
• Pure archival storage cost — on some database corpora we land near xz. On storage alone the difference often does not justify a format change, and we say so before the pilot rather than after.
• Single-stream general text — a well-tuned general compressor is hard to beat, and we frequently only tie.

The short checklist

1. Name the dataset. Public if possible.
2. State the baseline’s exact configuration, tuned in its favour.
3. Same codec set, comparable effort, both sides.
4. Prove lossless by hash — sha256(original) == sha256(reconstructed).
5. Reproduce from the shipped artefact, in CI, before publishing.
6. Publish the domains where you lose, in the same document.

None of this is novel; it is ordinary experimental hygiene borrowed from fields that settled these arguments long ago. It is worth writing down mainly because compression marketing routinely ignores all six, and because the discipline is what makes the numbers we do publish worth reading.