Newsroom
Method6 min read

The omission problem: no cryptography detects an event that was never recorded

By Dylan Wolpe

The short version

  • Hash chains, signatures, blockchains and WORM storage all protect entries that exist. None can detect an event that was never written, and this limit is mathematical rather than an implementation gap.
  • Completeness is an operational property, established by making the write path non-optional, monitoring its failures, and reconciling counts against an independent source.
  • Reconciliation is the strongest practical control: compare the log's own count against a number derived from a different system that had no involvement in writing it.
  • In practice audit trails fail far more often through silent write failure than through tampering, which means most integrity spending is aimed at the rarer risk.

We have now written twice that a hash chain cannot prove an event was recorded, each time as a caveat at the end. It deserves to be the subject rather than the footnote, because it is the limit that every product in this category shares and almost none of them state.

The shape of the gap

Integrity mechanisms answer one question: has what is here been altered? Hash chains, signatures, distributed consensus and write-once storage all answer it, with different costs and threat models.

None of them answers the other question: is everything here that should be?

This is not an implementation shortfall to be fixed in a later version. A record that was never created leaves no trace to detect. There is nothing for a hash to cover. A perfectly valid chain over an incomplete history is indistinguishable from a perfectly valid chain over a complete one.

Every mechanism in this category secures the entries you have. The failure mode that actually bites is the entry you never got.

Why this is the more common failure

Tampering with an audit log requires intent, access and a reason. Losing entries requires only an ordinary Tuesday:

  • The logging service was down for forty minutes and the application carried on.
  • A queue filled and dropped messages under load, precisely during the incident you will later want to reconstruct.
  • A code path was added that does not log, because logging was never in its acceptance criteria.
  • A batch job writes its audit entries at the end, and it crashed in the middle.
  • A retention policy quietly deleted the oldest entries a year earlier than intended.

Every one of these leaves the surviving log internally consistent and fully verifiable. Your integrity check passes. Your auditor is satisfied. The events are gone.

What actually establishes completeness

Since cryptography cannot, four operational controls do most of the work.

  1. Make the write path non-optional. If an action can succeed while its audit entry fails, you have chosen availability over auditability. That is sometimes the right choice and it should be a deliberate one, documented, rather than an accident of error handling.
  2. Alarm on the absence of logs. Most monitoring alerts on bad entries. Very little alerts on no entries. A silent logger looks exactly like a quiet period, and the gap is usually found weeks later by someone looking for something else.
  3. Number the entries. A monotonic sequence makes a gap visible after allocation. It cannot see an event that never entered the pipeline, but it converts a whole class of silent loss into an obvious hole.
  4. Reconcile against an independent source.The strongest control available. Compare the log’s count against a number derived from a system that had nothing to do with writing it, settled payments against ledger entries, badge-reader events against access lines, invoices against approvals. Its strength comes precisely from the second source being outside the first one’s failure domain.

What tamper-evidence is still for

None of this makes integrity mechanisms pointless, it clarifies what you are buying. Once the record exists, a hash chain guarantees it cannot be quietly revised later, and anchoring extends that to cover wholesale rebuilding. That is a genuine and necessary guarantee.

The correct mental model is two independent properties needing two different kinds of work: completeness is earned by engineering the write path, and integrity is earned by cryptography. A product can only sell you the second. Most of the risk lives in the first.

The question worth asking your own team

Not “is our audit log tamper-proof”. Instead: if our logging silently stopped for an hour, how would we find out, and how long would it take?

In most organisations the honest answer is that nobody would notice until someone went looking for a specific event and could not find it. That is a cheaper problem to fix than anything cryptographic, and fixing it removes considerably more risk.

Questions people ask about this

Can a blockchain prove my audit log is complete?

No. A blockchain establishes agreement about the entries that were submitted. If an event never reached it, the chain is valid and silent about the omission. This limit applies identically to hash chains, signed logs and WORM storage, it is not a weakness of one design.

How do you prove nothing is missing from a log?

You cannot prove it cryptographically. You establish it operationally: a write path that fails loudly rather than silently, alarms on logging outages, sequence numbering that makes gaps visible, and periodic reconciliation against a count derived from a system that had nothing to do with writing the log.

What is reconciliation in this context?

Comparing the log's own count of events against an independently derived count, payment records against the ledger's transaction entries, badge-reader events against access-log lines. A discrepancy means something is missing, and the strength of the control comes from the second source having no involvement in producing the first.

Does sequence numbering solve it?

It helps and does not solve it. Numbering makes a gap visible if a numbered entry is dropped after allocation. It cannot detect an event that never entered the pipeline and so was never assigned a number at all.

Related

More from the newsroom