Lookup elimination — at1 lookup
Wide attribute columns in a table are usually determined by a key: product_id → (name, category, supplier, price); zip → (city, state); device_id → (model, firmware). A general compressor re-encodes those attributes on every row; even a strong back-end keeps paying for the per-row reference. AT-1 Lookup discovers each categorical functional dependency A → B (every distinct value of A maps to exactly one B), stores the lookup map once per distinct key, drops the determined columns, and derives them on read.
Use it
# Auto-discover foreign-key / lookup dependencies and store the map once.
at1 lookup compress orders.csv --out orders.at1l
at1 lookup info orders.at1l # shows e.g. product_id -> {name, category, price}
at1 lookup decompress orders.at1l --out orders_out.csv # byte-exactNever-worse & verified
- Byte-exact — the map holds the exact original field bytes; the encoder rebuilds the file and compares before keeping the transform, and the container carries a SHA-256.
- Never-worse — if no eliminable dependency is found (or a raw store is smaller), it falls back to plain compression. Quoted/ragged CSV safely takes the raw path.
Related
This is the categorical case. For arithmetic derived columns (c = a + b), see derived-column elimination(at1 derived). Measured +23% vs the best whole-file backend on a realistic star-schema order table.