You could build this yourself. Here is what it would actually take.
By Dylan Wolpe
The short version
- Nothing in a verified, queryable, erasable archive is novel science. Hash chains, AES-GCM, columnar layout and Merkle proofs are all well-documented and a strong team can assemble them.
- The build is not the expensive part. Owning it for the ten-plus years of a retention obligation is, including the decoder still working after everyone who wrote it has left.
- Build when the archive is core to what you sell, when your data has structure no general tool models, or when regulation makes external dependencies unacceptable.
- Buy when it is infrastructure rather than product, and you would rather your engineers spend a decade on something your customers pay for.
Somebody senior always asks it, usually about twenty minutes in: could we just build this? The answer is yes, and a vendor who pretends otherwise is hoping you do not look closely. The interesting question is a different one.
What you would be building
None of it is exotic. Every piece is documented and has been for years.
| Component | Difficulty | The part that bites later |
|---|---|---|
| Columnar layout + typed encoding | Moderate | Every new type is a new codec, forever |
| Hash chain / Merkle structure | Low | Getting domain separation right the first time |
| Per-subject encryption + key vault | Moderate | Key custody discipline, not the crypto |
| Query pushdown over compressed blocks | High | Correctness under every predicate shape |
| Standalone offline verifier | Low | Keeping it working for a decade |
| Format versioning | Deceptively high | Year-one archives must decode in year ten |
A strong team gets to something working in a few months. We are not going to pretend that is heroic, it is a normal infrastructure project.
The build is a few months. The ownership is the retention period, and the retention period is the entire point of the thing you are building.
The costs that arrive after launch
Backwards compatibility becomes permanent. This is the one teams underestimate most. An archive written today must decode byte-exactly in ten years, through every format revision you make in between. That requires a corpus of historical archives that only grows, run against every build, forever. Miss it once and you discover the gap during a restore, which is the worst possible moment.
The verifier has to outlive the team. Whoever checks an archive in 2036 will not be the person who wrote the checker. That means real documentation of the format, a specification someone could implement from, rather than the code being the specification. Most internal projects never produce this, because there is never a quarter in which writing it is the most urgent thing.
Edge cases are found by data, not by design. Every unusual encoding, every malformed record, every column that is 99% null and then suddenly is not. These surface over years of contact with real data, and each one is a bug found by someone who needed the archive to work today.
Someone owns it while nobody is looking. Infrastructure that works is infrastructure nobody is staffed for. Three years in, the original authors have moved on, and the archive format is maintained by whoever most recently touched it by accident.
When building is right
- The archive is your product. If customers buy the storage layer, owning it is strategy rather than cost.
- Your data has structure nothing general models. A domain-specific encoder can beat a general one substantially, and if that gap is your competitive advantage it belongs in house.
- The dependency is genuinely unacceptable. Some regulated environments cannot take an external component in this path, and that is a legitimate constraint rather than a preference.
- You have a team that will still exist in five years. Rarer than it sounds, and the single best predictor of whether the build succeeds.
Our honest position
We would rather you bought it, obviously. But the decision that actually matters is whether this is core or plumbing for you. If it is plumbing, a decade of your engineers’ attention is a high price for avoiding a licence, and if it is core, no vendor should be holding it, including us.
If you build, do read the piece on outliving the vendor, because every argument in it applies with equal force to outliving your own team. The internal project that nobody documented is the same failure as the vendor who disappeared, arriving from a different direction.
Questions people ask about this
Could we build a verified, queryable archive in house?
Yes. Every primitive involved is standard and well documented. A capable team can produce a working system in a few months. The difficulty is not the initial build; it is maintaining a format, a decoder and a verifier for the full length of the retention obligation, including after the original authors have moved on.
What is the hardest part to get right?
Long-horizon compatibility. An archive written in year one must still decode byte-exactly in year ten, across format revisions, dependency upgrades and platform changes. That requires a compatibility test corpus that grows forever and gates every release, which is an ongoing discipline rather than a piece of code.
When does building make more sense than buying?
When the archive is part of your product rather than your plumbing, when your data has domain structure no general-purpose tool models well, or when regulation genuinely forbids the dependency. In those cases the ownership cost is buying you something strategic rather than just saving a licence fee.