What a file size tells you about the system that produced it
By Dylan Wolpe
The short version
- Compression works by finding structure and describing it briefly, so the size of the output is a measurement of how much structure the input contained.
- That makes a compressor a general-purpose structure detector: it will register any regularity, whether or not the regularity is the one you were looking for.
- The measurement is only meaningful relative to a control encoded through the identical pipeline, because artefacts of the encoding are indistinguishable from findings without one.
- It is a screening instrument rather than a specialist one, good at noticing that something is there, weak at saying what, and routinely beaten by a purpose-built model once you know what to look for.
A compressor is usually thought of as a storage tool. It is more accurately a measuring instrument that happens to leave a smaller file behind as a by-product, and once you see it that way a set of otherwise unrelated problems turn out to be the same problem.
The idea in one paragraph
Compression works by finding regularity and describing it more briefly than the raw data. Data with a lot of structure compresses well because the structure can be stated once. Data with none compresses barely at all, because there is nothing to say except the data itself.
So the compressed size is not just an outcome. It is a reading: how much structure is in here? And a reading can be compared, against a baseline, against a control, against the same source last month.
The file got smaller is a storage fact. The file got smaller than it used to is a measurement, and measurements are what you can act on.
Three questions that turn out to be one
| The question | What is being measured |
|---|---|
| Is this lottery fair? | Structure where there should be none |
| Is this bearing wearing out? | Structure changing where it should be steady |
| Did this data source change silently? | Structure changing where nothing was announced |
These sit in different industries and are usually solved by different specialists with different tools. Framed as measurement they are one operation, take a reading, compare it to what it should be, which is the practically useful part of the idea rather than a philosophical one.
The detector does not know what you are looking for
This is the property that makes it powerful and the property that makes it dangerous, and they are the same property. A compressor registers any regularity. It has no notion of which regularities are interesting.
We demonstrated this against ourselves. Pointed at national lottery results, our detector reported a large, unambiguous bias. The bias was real structure and it was not a rigged lottery, the game had changed format part-way through the history we had pooled, so a set of balls was impossible for the first several years. The instrument reported exactly what was there. The experiment was asking a question that could not distinguish “rigged” from “the rules changed”. That is written up in full, wrong answer included, in the PowerBall audit.
The generalisation is uncomfortable: the more sensitive your instrument, the more reliably it will find your own pipeline. Schema migrations, firmware updates, sensor swaps, a vendor changing their export format, all of these look exactly like discoveries, and they arrive with high confidence.
Which is why the control is not optional
The discipline that makes this usable is comparing against data that went through the identical pipeline and is known not to contain the effect. Synthetic draws from a fair generator. A period when the machine was known-healthy. Data from before a change you are confident about.
Every artefact of encoding, formatting and collection appears in both, so it cancels. Whatever difference survives is a property of the world. Without a control you are not measuring the system, you are measuring the system plus your own tooling, and you have no way to tell which contributed what.
Why we care about this at all
Because it explains what the company is actually built on. We sell storage and verification products, and those rest on modelling data well enough to encode it briefly. Modelling data that well produces a measurement as a side effect, and the measurement is often worth more than the bytes saved.
It also sets the boundary of the claim. Structure-aware compression wins where data is rule-governed and floors honestly where it is not, which is the same statement as the detector finding nothing in a fair lottery. A tool that reported a large win on random data would not be a better compressor. It would be a broken instrument, and you would want to know that before trusting anything else it told you.
Questions people ask about this
How can a compressor detect anomalies?
Compression finds patterns and encodes them briefly, so a file's compressed size reflects how much structure it holds. Data that suddenly compresses better or worse than its baseline has changed in structure, which is often the earliest observable sign that the system producing it has changed.
What does compressed size actually measure?
An upper bound on the information content of the data under the model your compressor implements. It is not an absolute quantity, a different compressor implies a different model and a different number, which is why comparisons are only valid within one pipeline.
Why do you always need a control?
Because the detector responds to all structure, including structure created by your own encoding, formatting or data collection. Comparing against synthetic data pushed through the identical pipeline cancels those artefacts, so whatever difference survives is a property of the world rather than of your file format.
Is this better than a purpose-built anomaly model?
No, and it is not trying to be. A model trained for a specific fault beats it on that fault. The advantage is that it needs no labels, no training and no prior idea of what you are looking for, it is a screening instrument that tells you where to point the specialist.