Compute on compressed data, skip the tax
An ML pipeline pays to decompress the same columns thousands of times. AT-1 stores columns as block-compressed segments with a per-block zone map, then extracts features — counts, sums, means, min/max, and range-filtered aggregates — touching only the blocks whose zone map overlaps the predicate. Results are byte-identical to a full scan; a selective query reads a fraction of the bytes.
- 128×
- fewer bytes read on a clustered predicate (touched 1/128 blocks)
- byte-identical
- results equal a full decompress-and-scan, exactly
- 0 bytes
- decoded for summary features, computed from zone maps alone
- SHA-checked
- every block verified on decode; tamper refused
Extract features, read only what you need
at1 features build ticks.csv -o ticks.at1feat at1 features extract ticks.at1feat --where ts:1700262278:1700265482 # WHERE ts in [...] (touched 1/128 blocks, read 4,559 of # 585,272 bytes = 128.4x less): # qty: count=1601 sum=397013 mean=247.98 min=1 max=500 at1 features extract ticks.at1feat # summary: zone-maps only, 0 bytes decoded
Zone-map pushdown
Blocks whose min/max can’t overlap the filter are skipped without a single byte read, the reduction scales with how clustered your predicate is.
Exact, and verified
Features are byte-identical to a full scan; every decoded block is SHA-checked against the manifest, so tampering is refused.
Honest boundary
The reduction depends on physical clustering. A random-scattered predicate degrades toward a full scan, the tool reports the exact bytes touched so you always see where you sit.
Billed per extraction, first 10,000/month free. See pricing.
Decompressing to compute is a habit, not a requirement
The usual shape is to unpack a file into memory, compute over it, and throw the unpacked copy away. For a sum, a count or a feature extraction, most of that work is spent rebuilding bytes you are about to discard, and the peak memory of the job is set by the uncompressed size of data you never needed in that form.
Because the container is blocked and indexed, an aggregate touches each block once and a range filter skips the blocks that cannot contain a match. The memory the job needs is set by the block, not by the dataset, which is the difference between a machine that can run the job and one that cannot.
The honest boundary
This suits operations that read broadly and return something small: sums, counts, ranges, feature extraction. An operation that genuinely needs every value in memory at once, a full sort or an arbitrary join, gets no benefit and should decompress. We would rather you knew which of those two your workload is before you build on it.