Dedup tier — at1 dedup

A general compressor only removes redundancy it can see inside its window. Across many files — daily backups, document revisions, container layers, model checkpoints, log rotations — the same blocks recur but live in different files, so per-file compression never sees them. AT-1 Dedup chunks every file at a content-defined boundary (a rolling hash over the bytes), so inserting or editing a few bytes shifts only the chunk it lands in — every other chunk keeps its identity and dedups against the pool. Unique chunks are stored once and entropy-coded; each file becomes a thin manifest of references.

Use it

# Dedup-pack a folder of files/versions into one verified archive.
at1 dedup pack   ./backups        --out daily.at1d
at1 dedup info   daily.at1d        # unique chunks, dedup ratio, stored size
at1 dedup unpack daily.at1d --out ./restored   # byte-exact + per-file SHA-256 verified

What it’s for

  • Backups & snapshots — N daily copies that mostly repeat.
  • Document / dataset revisions — versioned exports, edited files.
  • Container & image layers, model checkpoints, log rotations.

Verified by construction

  • Byte-exact reconstruction, and every original file is SHA-256-checked on unpack.
  • Insertion-resilient — content-defined boundaries re-sync after an edit, so a 1-byte change doesn’t re-store the whole file.

Honest scope

On a small corpus that fits a strong compressor’s window, whole-corpus xz ties dedup on ratio. Dedup’s ratio win shows up beyond the window (large corpora a streaming compressor can’t span) — and it uniquely gives you per-file random access, incremental add, and per-file verified integrity that a single concatenated stream can’t. Measured −91% vs compressing each revision independently (the real backup scenario).

← CLI reference