Method13 October 20265 min read

Silent corruption does not announce itself

The short version

Most pipeline verification checks that the job succeeded and that row counts match. Neither notices a transformation that altered values while preserving count and schema.
A content hash taken at the boundary and re-checked after turns 'the job reported success' into 'the bytes are identical', which is a different and much stronger claim.
Where a pipeline must legitimately transform data, hash each stage's input and output so a later discrepancy can be localised to a stage rather than to the whole system.
The cost is a hash per stage. The alternative is discovering the problem downstream and having no way to establish when it started.

Ask how anyone knows their nightly load moved the data faithfully and the answer is that the job went green and the row counts matched. Both statements are true of a pipeline that truncated a field on 0.4% of records.

What the usual checks cannot see

Three checks are near-universal, and each has a specific blind spot:

Job status. Reports whether the code raised an exception. A transformation that quietly mangles values is not an exception; it is the code doing what it was told.
Row counts. Confirms nothing was dropped. Says nothing about what happened inside the rows that arrived.
Schema validation. Confirms the shape. A truncated string is still a string; a re-rounded float is still a float.

Green job, matching counts, valid schema, and a subset of values silently different. All three checks are working exactly as designed.

The failures that actually occur

None of these are exotic. All of them complete successfully:

A column defined narrower downstream, truncating values past the limit.
Character encoding mismatch mangling any non-ASCII text.
A float re-rounded on the way through, changing sums by a fraction.
Timezone conversion applied twice, shifting a subset of timestamps.
Nulls coerced to empty strings, or to zero.
A library upgrade changing default precision.

The last is the nastiest, because nothing in your code changed. A dependency’s default moved, and the pipeline began producing subtly different output from identical input, on a date nobody recorded.

Hash the boundary

Where a stage is meant to be faithful, a copy, a compression, a format conversion that should round-trip, the check is exact and cheap:

  h_before = sha256(source_bytes)
  ...move / compress / convert...
  h_after  = sha256(reconstructed_bytes)

  assert h_before == h_after

An equality test, not a heuristic. It either matches or the data changed, and there is no third answer to argue about in a post-mortem.

This is what “verified lossless” means as an operational practice rather than a format property: not that the format can reproduce the input, but that on this run, for this data, it demonstrably did. Every archive we write carries the hash of what went in, so the round trip is checkable by whoever holds the file, whenever they like, without us.

Pipelines that legitimately transform

Most stages are not faithful moves, aggregation, joins and enrichment all change the data on purpose, so input and output hashes should differ.

Hash both sides of each stage anyway:

  stage 1   in: a1   out: b1
  stage 2   in: b1   out: c1        <- b1 must equal stage 1's out
  stage 3   in: c1   out: d1        <- c1 must equal stage 2's out

You cannot assert equality across a transformation. You can assert that stage 3 received exactly what stage 2 produced.

When a downstream number looks wrong three weeks later, you can establish which stage first saw something unexpected, instead of forming a theory about a system with eleven moving parts and no record of what any of them received.

Where to start

Identify which stages are supposed to be faithful. Usually more than people assume.
Hash each side of those and assert equality.
For transforming stages, record both hashes without asserting.
Keep the hashes as long as you keep the data, so localisation is possible later.

The cost is a hash per stage, which is negligible next to the work the stage is already doing. The alternative is finding out from a customer, and having no way to establish when it started or how much is affected, which is the position most organisations are in and have not noticed because nothing has gone wrong loudly enough yet.

Questions people ask about this

How do you verify data was not altered by a pipeline?

Compute a content hash before and after and compare them. If the stage is meant to be a faithful move, a copy, a compression, a format conversion that should be reversible, the hashes must match exactly. Row counts and job status do not establish this; only comparing content does.

What is silent data corruption?

Any change to data that occurs without an error being raised: a truncated field, a character encoding mangled, a float re-rounded, a timezone applied twice. The job completes, the schema validates, the counts match, and a subset of values are wrong.

How do you verify pipelines that legitimately transform data?

Hash the input and output of each stage separately and record both. You cannot expect input and output to match when a transformation is intended, but you can prove that a given stage received exactly what the previous stage produced, which localises any later discrepancy to one stage.