AT-1 SDK

One encoder, a decoder that runs everywhere, and a query API. The encoder stays server-side; the decoder is small, embeddable, and available in Python, C, Go, Rust, and WebAssembly from a single fuzz-hardened C core via a stable ABI.

The golden rule everywhere: decode(compress(x)) == x, byte-for-byte, or the encoder falls back to a non-inferior raw encoding. Malformed input is always rejected cleanly, the decoder never crashes the host.

Python, the full product (encode, decode, query)

npm i -g @tinyfiles/cli               # the at1 CLI as a self-contained binary, no Python required

# encode / decode (CLI or module)
#   at1 compress columnar data.csv data.at1
#   at1 decompress data.at1 data.out
import subprocess
subprocess.run(["at1", "compress", "qcolumnar", "trades.csv", "trades.at1"])

# query WHILE compressed (reads only the blocks a predicate can't rule out)
from at1reader import AT1Reader
r = AT1Reader("trades.at1")
print(r.schema)                                        # columns, types, row-groups
rows, stats = r.time_range(5, t0, t1, select=[0, 1, 5])   # range pushdown on a time col
rows, stats = r.scan(where={6: ("=", "True")}, select=[0])  # equality + projection
# stats: groups_scanned/skipped, blocks_read, bytes_read, matched

Fast native decode (no Python codec modules) via the C library:

from at1decode import decode, decode_with_stats, version   # bindings/python/at1decode.py (ctypes over the ABI)
original = decode(open("data.at1", "rb").read())            # bytes -> bytes, raises AT1Error on bad input
original, stats = decode_with_stats(open("data.at1", "rb").read())
# stats: {original_bytes, compressed_bytes, io_bytes}  (decode is unmetered; counters are informational)
print(version())

Query from the engine you already run (11 engines)

One engine-agnostic decode core (at1_block.c) feeds every engine, directly in C, or through Apache Arrow. All verified live:

• DuckDB, native C extension (duckdb_at1/capi/) + Python adapter; SELECT … FROM read_at1('a.at1').
• SQLite, native C virtual table: CREATE VIRTUAL TABLE t USING at1('a.at1', …).
• PostgreSQL, native C foreign data wrapper; .at1 as foreign tables.
• ClickHouse, Spark, via the Arrow export; Trino, Presto, Flink, via the Postgres/JDBC bridge.
• Polars / pandas / Dask, import at1_arrow, zero new code.

import at1_arrow
df  = at1_arrow.to_polars("trades.at1")       # or to_pandas / to_arrow
at1_arrow.write_ipc("trades.at1", "trades.arrow")   # universal handoff to any Arrow engine

Full matrix + reproduce commands: docs/ADAPTERS.md.

AI agents, the AT-1 MCP server

An MCP server (mcp_server/) gives Claude Desktop, Cursor, ChatGPT and IDEs four tools, at1_compress, at1_info, at1_query, at1_verify_integrity, so an agent can compress, inspect, query, and tamper-check AT-1 archives directly. See mcp_server/README.md for editor configs.

C, the ABI everything else is built on

#include "at1_decode.h"        // c_decoder/at1_decode.h
// link with -llzma -lzstd (or an embedded decode-only profile)
uint8_t *out; size_t out_len;
int rc = at1_decode_buffer(in, in_len, &out, &out_len);   // 0=OK, 2=corrupt, 3=backend
if (rc == AT1_OK) {
    /* use out[0..out_len) */
    at1_free(out);                                        // free the malloc'd buffer
}                                                         // on error *out is untouched

at1_decode_buffer allocates the output; free it with at1_free. The decoder is bounds-checked and never calls exit() or aborts the host, on bad input it returns AT1_ERR_CORRUPT / AT1_ERR_BACKEND and leaves *out untouched. Build the shared lib: cd c_decoder && make (or the one-liner in bindings/README.md).

WebAssembly, decode in the browser / at the edge

import { decode, version } from "@tinyfiles/decoder";   // c_decoder/wasm (at1.mjs)
const original = await decode(at1Bytes);      // Uint8Array -> Uint8Array (async: wasm init)
console.log(await version());                 // malformed input throws, never crashes the host

Full backend + codec coverage (xz, zstd, stored, incl. qcolumnar) with liblzma cross-compiled in. See c_decoder/wasm/README.md.

Go (cgo)

import "github.com/AT1-Tinyfiles/AT-1-Public/bindings/go/at1"   // cgo, CGO_ENABLED=1

original, err := at1.Decode(at1Bytes)         // []byte -> []byte
if err != nil { /* corrupt/backend input rejected, host never crashes */ }
out, stats, err := at1.DecodeWithStats(at1Bytes)   // stats: OriginalBytes/CompressedBytes/IOBytes
_ = at1.Version()

Needs the liblzma + libzstd dev libraries and CGO_ENABLED=1 with a C compiler on PATH. Decode returns an error (never a crash) for malformed input.

Rust (FFI)

// bindings/rust/at1, safe wrapper over the C ABI
let at1_bytes = std::fs::read("data.at1")?;
let original = at1::decode(&at1_bytes)?;                 // Vec<u8>, At1Error on bad input
let (out, stats) = at1::decode_with_stats(&at1_bytes)?;  // stats: original_bytes/compressed_bytes/io_bytes
println!("{}", at1::version());

A safe wrapper over the C ABI: decode yields a Vec<u8> or an At1Error (Corrupt / Backend / Other), the process is never aborted.

Node (native N-API)

// @tinyfiles/node is an EXPERIMENTAL native N-API addon, it builds from source on
// install (node-gyp), so it needs a C++ toolchain. For a zero-build Node/edge path,
// prefer @tinyfiles/decoder (WASM) below, which installs with no toolchain.
const at1 = require("@tinyfiles/node");       // bindings/node (native N-API addon)
const original = at1.decode(at1Bytes);        // Buffer|Uint8Array -> Buffer
const { original: buf, stats } = at1.decodeWithStats(at1Bytes);   // {originalBytes, compressedBytes, ioBytes}
console.log(at1.version());                   // all backends (xz, zstd, stored) + all codecs

The N-API addon dynamically loads the decoder shared library at startup, so this native path supports all backends (xz, zstd, stored) and all codecs, unlike the WASM build, which today covers the zstd/stored backends. Point it at a specific library with AT1_DECODER_LIB. See bindings/node/README.md.

Format versioning & conformance

• Container version is the 4-byte magic: AT1\x02 (whole-file) and AT1\x03 (streaming). A decoder dispatches on it; new container revisions bump the trailing byte. Codec ids are 0–12 (RAW = 255); backend ids are 0 = xz, 1 = zstd, 2 = stored.
• Spec: c_decoder/AT1_FORMAT_SPEC.md.
• Conformance: c_decoder/conformance/manifest.json lists every vector with the SHA-256 of the exact output a conforming decoder must emit. Validate any decoder:

cd c_decoder/conformance
python run_conformance.py "../at1_decode {in} {out}"              # your decoder here
python run_conformance.py "python ../../at1.py decompress {in} {out}"

Both the native C decoder and the Python reference pass 11/11. Every binding above, Python, C, WASM, Go, Rust, Node, is verified against the same shared conformance vectors, so a decode on any surface reproduces the original bytes exactly or rejects the input cleanly (see the per-surface counts in the table at the top of this page).

Semantic versioning

For the current published version, run npm view @tinyfiles/cli version. Decoders accept any container whose magic they recognize; unknown codec/backend ids are rejected with AT1_ERR_CORRUPT/_BACKEND rather than mis-decoded. Backward compatibility target: a decoder never silently produces wrong bytes, it either reproduces the input exactly or errors.