AT-1 SDK

One encoder, a decoder that runs everywhere, and a query API. The encoder stays server-side; the decoder is small, embeddable, and available in Python, C, Go, Rust, and WebAssembly from a single fuzz-hardened C core via a stable ABI.

The golden rule everywhere: decode(compress(x)) == x, byte-for-byte, or the encoder falls back to a non-inferior raw encoding. Malformed input is always rejected cleanly — the decoder never crashes the host.

Python — the full product (encode, decode, query)

npm i -g @tinyfiles/cli               # the at1 CLI as a self-contained binary — no Python required

# encode / decode (CLI or module)
#   at1 compress columnar data.csv data.at1
#   at1 decompress data.at1 data.out
import subprocess
subprocess.run(["at1", "compress", "qcolumnar", "trades.csv", "trades.at1"])

# query WHILE compressed (reads only the blocks a predicate can't rule out)
from at1reader import AT1Reader
r = AT1Reader("trades.at1")
print(r.schema)                                        # columns, types, row-groups
rows, stats = r.time_range(5, t0, t1, select=[0, 1, 5])   # range pushdown on a time col
rows, stats = r.scan(where={6: ("=", "True")}, select=[0])  # equality + projection
# stats: groups_scanned/skipped, blocks_read, bytes_read, matched

Fast native decode (no Python codec modules) via the C library:

from at1decode import decode, decode_with_stats, version   # bindings/python/at1decode.py (ctypes over the ABI)
original = decode(open("data.at1", "rb").read())            # bytes -> bytes, raises AT1Error on bad input
original, stats = decode_with_stats(open("data.at1", "rb").read())
# stats: {original_bytes, compressed_bytes, io_bytes}  (decode is unmetered; counters are informational)
print(version())

Query from the engine you already run (11 engines)

One engine-agnostic decode core (at1_block.c) feeds every engine — directly in C, or through Apache Arrow. All verified live:

• DuckDB — native C extension (duckdb_at1/capi/) + Python adapter; SELECT … FROM read_at1('a.at1').
• SQLite — native C virtual table: CREATE VIRTUAL TABLE t USING at1('a.at1', …).
• PostgreSQL — native C foreign data wrapper; .at1 as foreign tables.
• ClickHouse, Spark — via the Arrow export; Trino, Presto, Flink — via the Postgres/JDBC bridge.
• Polars / pandas / Dask — import at1_arrow, zero new code.

import at1_arrow
df  = at1_arrow.to_polars("trades.at1")       # or to_pandas / to_arrow
at1_arrow.write_ipc("trades.at1", "trades.arrow")   # universal handoff to any Arrow engine

Full matrix + reproduce commands: docs/ADAPTERS.md.

AI agents — the AT-1 MCP server

An MCP server (mcp_server/) gives Claude Desktop, Cursor, ChatGPT and IDEs four tools — at1_compress, at1_info, at1_query, at1_verify_integrity — so an agent can compress, inspect, query, and tamper-check AT-1 archives directly. See mcp_server/README.md for editor configs.

C — the ABI everything else is built on

#include "at1_decode.h"        // c_decoder/at1_decode.h
// link with -llzma -lzstd (or an embedded decode-only profile)
uint8_t *out; size_t out_len;
int rc = at1_decode_buffer(in, in_len, &out, &out_len);   // 0=OK, 2=corrupt, 3=backend
if (rc == AT1_OK) {
    /* use out[0..out_len) */
    at1_free(out);                                        // free the malloc'd buffer
}                                                         // on error *out is untouched

at1_decode_buffer allocates the output; free it with at1_free. The decoder is bounds-checked and never calls exit() or aborts the host — on bad input it returns AT1_ERR_CORRUPT / AT1_ERR_BACKEND and leaves *out untouched. Build the shared lib: cd c_decoder && make (or the one-liner in bindings/README.md).

WebAssembly — decode in the browser / at the edge

import { decode, version } from "@tinyfiles/decoder";   // c_decoder/wasm (at1.mjs)
const original = await decode(at1Bytes);      // Uint8Array -> Uint8Array (async: wasm init)
console.log(await version());                 // malformed input throws, never crashes the host

Full backend + codec coverage (xz, zstd, stored, incl. qcolumnar) with liblzma cross-compiled in. See c_decoder/wasm/README.md.

Go (cgo)

import "github.com/FelixKramer/at1/bindings/go/at1"   // cgo, CGO_ENABLED=1

original, err := at1.Decode(at1Bytes)         // []byte -> []byte
if err != nil { /* corrupt/backend input rejected, host never crashes */ }
out, stats, err := at1.DecodeWithStats(at1Bytes)   // stats: OriginalBytes/CompressedBytes/IOBytes
_ = at1.Version()

Needs the liblzma + libzstd dev libraries and CGO_ENABLED=1 with a C compiler on PATH. Decode returns an error (never a crash) for malformed input.

Rust (FFI)

// bindings/rust/at1 — safe wrapper over the C ABI
let at1_bytes = std::fs::read("data.at1")?;
let original = at1::decode(&at1_bytes)?;                 // Vec<u8>, At1Error on bad input
let (out, stats) = at1::decode_with_stats(&at1_bytes)?;  // stats: original_bytes/compressed_bytes/io_bytes
println!("{}", at1::version());

A safe wrapper over the C ABI: decode yields a Vec<u8> or an At1Error (Corrupt / Backend / Other) — the process is never aborted.

Node (native N-API)

const at1 = require("@at1/node");             // bindings/node (native N-API addon)
const original = at1.decode(at1Bytes);        // Buffer|Uint8Array -> Buffer
const { original: buf, stats } = at1.decodeWithStats(at1Bytes);   // {originalBytes, compressedBytes, ioBytes}
console.log(at1.version());                   // all backends (xz, zstd, stored) + all codecs

The N-API addon dynamically loads the decoder shared library at startup, so this native path supports all backends (xz, zstd, stored) and all codecs — unlike the WASM build, which today covers the zstd/stored backends. Point it at a specific library with AT1_DECODER_LIB. See bindings/node/README.md.

Format versioning & conformance

• Container version is the 4-byte magic: AT1\x02 (whole-file) and AT1\x03 (streaming). A decoder dispatches on it; new container revisions bump the trailing byte. Codec ids are 0–12 (RAW = 255); backend ids are 0 = xz, 1 = zstd, 2 = stored.
• Spec: c_decoder/AT1_FORMAT_SPEC.md.
• Conformance: c_decoder/conformance/manifest.json lists every vector with the SHA-256 of the exact output a conforming decoder must emit. Validate any decoder:

cd c_decoder/conformance
python run_conformance.py "../at1_decode {in} {out}"              # your decoder here
python run_conformance.py "python ../../at1.py decompress {in} {out}"

Both the native C decoder and the Python reference pass 11/11. Every binding above — Python, C, WASM, Go, Rust, Node — is verified against the same shared conformance vectors, so a decode on any surface reproduces the original bytes exactly or rejects the input cleanly (see the per-surface counts in the table at the top of this page).

Semantic versioning

The package version is 0.2.2. Decoders accept any container whose magic they recognize; unknown codec/backend ids are rejected with AT1_ERR_CORRUPT/_BACKEND rather than mis-decoded. Backward compatibility target: a decoder never silently produces wrong bytes — it either reproduces the input exactly or errors.