Find near-duplicates and novelty — by compression distance.
at1 ncd uses normalized compression distance to measure how similar two files are without parsing them — a code-clone / near-duplicate scan and a clause/document novelty triage. It sees surface structure, not meaning.
Two detectors, one distance
Code-clone & near-duplicate scan
Scan a tree and surface files (or functions) that are near-copies of each other by compression distance — copy-paste clones, forked-and-diverged files, redundant assets — without a language parser or an AST.
Clause & document novelty triage
Given a corpus of prior documents, score how novel a new one is: which clauses or paragraphs closely echo something already on file, and which are genuinely new — a fast first pass before a human reads closely.
Parser-free, format-agnostic
NCD works on raw bytes, so the same detector runs on source, contracts, logs, or binaries with no per-format setup. It's a triage instrument that points a reviewer at the pairs worth looking at.
One command surface
# normalized compression distance: 0 = near-identical, 1 = unrelated at1 ncd distance a.py b.py # -> 0.07 (near-duplicate) at1 ncd clones ./src --threshold 0.2 # find code-clone / near-dup clusters at1 ncd novelty new.txt --corpus ./prior/ # which clauses echo prior docs, which are new
Honest scope
NCD is a surface-similarityinstrument: it finds byte-level closeness, not semantic equivalence. Two files that say the same thing in different words read as far apart, and two unrelated files that share boilerplate read as close — so it’s a triage pass that points a human at the pairs worth reviewing, not a plagiarism or equivalence verdict. It is calibrated on synthetic and exemplar data, with broad real-corpus validation as the explicit graduation gate. For “what kind of file is this?” see at1 origin; for “how rule-governed is it?” see at1 determinism.