Dedup tier, at1 dedup

A general compressor only removes redundancy it can see inside its window. Across many files, daily backups, document revisions, container layers, model checkpoints, log rotations, the same blocks recur but live in different files, so per-file compression never sees them. AT-1 Dedup chunks every file at a content-defined boundary (a rolling hash over the bytes), so inserting or editing a few bytes shifts only the chunk it lands in, every other chunk keeps its identity and dedups against the pool. Unique chunks are stored once and entropy-coded; each file becomes a thin manifest of references.

Use it

# Dedup-pack a folder of files/versions into one verified archive.
at1 dedup pack   ./backups        --out daily.at1d
at1 dedup info   daily.at1d        # unique chunks, dedup ratio, stored size
at1 dedup unpack daily.at1d --out ./restored   # byte-exact + per-file SHA-256 verified

What it’s for

  • Backups & snapshots, N daily copies that mostly repeat.
  • Document / dataset revisions, versioned exports, edited files.
  • Container & image layers, model checkpoints, log rotations.

Verified by construction

  • Byte-exact reconstruction, and every original file is SHA-256-checked on unpack.
  • Insertion-resilient, content-defined boundaries re-sync after an edit, so a 1-byte change doesn’t re-store the whole file.

Honest scope

On a small corpus that fits a strong compressor’s window, whole-corpus xz ties dedup on ratio. Dedup’s ratio win shows up beyond the window (large corpora a streaming compressor can’t span), and it uniquely gives you per-file random access, incremental add, and per-file verified integrity that a single concatenated stream can’t. Measured −91% vs compressing each revision independently (the real backup scenario).

← CLI reference