Record store — at1 zdict

Logs, events, telemetry messages and database rows are individually tiny — tens to hundreds of bytes. A general compressor needs a warm-up window to build statistics, so compressing each record alone barely shrinks it — yet you must compress them alone if you want to fetch record N without inflating the whole batch. AT-1’s record store trains a dictionary on a sample of the records, so every record is compressed primed with the corpus’s common substrings: each stays independently decompressable (true random access) while shrinking far more than naive per-record compression.

Use it

# Dictionary-pack newline-delimited records (NDJSON / log lines) into a random-access store.
at1 zdict pack   events.ndjson  --out events.at1z
at1 zdict get    events.at1z  1234     # decompress ONLY record #1234 (no full unpack)
at1 zdict info   events.at1z
at1 zdict unpack events.at1z  --out events_out.ndjson   # all records, SHA-256 verified

# RAG / prompt corpora: records that contain newlines, packed with --json-lines
at1 zdict pack prompts.jsonl --json-lines --out prompts.at1z   # ~3x smaller, per-prompt random access
at1 zdict get  prompts.at1z 42                                  # fetch ONLY prompt #42

What it’s for

  • Log & event archives you still need to seek into by offset/index.
  • Row stores where any single row must come back without unpacking the file.
  • Telemetry / message queues persisted for replay and audit.

RAG & prompt corpora

Prompt logs and RAG contexts are a natural fit: they repeat a shared system prompt and reuse the same snippets across thousands of records. at1 zdict pack --json-lines packs a corpus of prompts / contexts — records that themselves contain newlines — into a trained-dictionary, per-record random-access store, so the shared boilerplate dedups across the whole corpus instead of being re-compressed in every record. Measured ~3× smaller than compressing each prompt on its own, with per-prompt random access intact— fetch prompt N without inflating the corpus. (If you don't need random access, batch-compressing the whole corpus gives a bigger multiple — this is the random-access tier.)

Verified

  • Per-record length-bound checks, plus a whole-store SHA-256 integrity trailer.
  • get i decompresses exactly one record — the random-access guarantee.

Honest scope

If you never need per-record access, just batch-compress the whole file — that wins on raw ratio. This is the random-access tier: measured ~47% smaller than naive per-record compression with random access intact (the dictionary amortizes to a few bytes per record at scale).

← CLI reference