Lakehouse cold tier — `at1 lakehouse`

Store a Delta, Iceberg, or Parquet table’s data files as AT-1 — about 27% smaller than Parquet+zstd, byte-for-byte lossless — without changing the table format, the catalog, or your engines. See /lakehouse for the measured results.

What it changes, and what it doesn’t

It changes the storage of each active data file: part.parquet becomespart.parquet.at1. It does not touch the Delta _delta_log or the Iceberg metadata/ — those still name *.parquet. So the catalog and your table definitions are unchanged; you are swapping the bytes at rest.

For Delta specifically it is format-aware: only the active working set (the add-minus-remove files of the current version) is tiered, resolved via delta-rs or by replaying the commit log; tombstoned files are left alone so time-travel reads keep working, and every action is written to a hash-chained ledger so the tiering is tamper-evident and re-runs are idempotent.

Usage

# the first run downloads the compiled engine (bundles Arrow, ~120MB), SHA-256 verified, cached
at1 login --key <API_KEY>                     # tiering is an encode -> needs a free account

at1 lakehouse status    ./table_or_dir         # table type, files, and the saving
at1 lakehouse tier      ./table_or_dir         # store each data file as .parquet.at1
at1 lakehouse tier      ./table --delete-original   # remove the .parquet once tiered
at1 lakehouse rehydrate ./table_or_dir         # materialise .parquet.at1 back to .parquet
at1 lakehouse serve     ./dir --host 0.0.0.0 --port 8088   # gateway: read .at1 as Parquet

Two ways engines read it back

Gateway (Spark, Trino, DuckDB, Databricks). at1 lakehouse serve runs an HTTP gateway that materialises Parquet on the fly from the .at1files, with Range support so an engine reads the footer and row-group byte ranges. Point the engine at the gateway URL as an object-store path. This is genuine query-in-place — no materialised second copy.
Materialise into S3 (Snowflake). Snowflake allowlists storage providers and will not read a custom endpoint, so for Snowflake you materialise the Parquet into a recognised store (S3) and point a stage at it. Verified byte-identical against real NYC-TLC.

Verified

On official NYC-TLC (1M rows, 19 columns): AT-1 stored 27.3% smaller than Parquet+zstd-9, lossless both directions. Snowflake (external S3 stage) and Databricks (Unity Catalog volume) each returned byte-identical results for count, sum(fare_amount),avg(trip_distance), a filtered count and count(distinct PULocationID)between the original Parquet and the AT-1-materialised Parquet. Reproduce with the scripts in benchmarks/experiments/ (LAKEHOUSE_COLDTIER_RESULTS.md).

Billing

tier is an encode— it writes your table into AT-1 — so it needs a connected account and meters compressed bytes on the same per-TB usage meter as the rest of AT-1. status, serve, rehydrate and all reading are free and need no account.

Delivery & limits

Compiled, downloaded on first use. The engine bundles Apache Arrow (~120MB), too heavy for the base CLI, so at1 lakehouse downloads the compiled binary from our public release, verifies its SHA-256, and caches it under ~/.at1/lakehouse. No Python, no source. Pre-place it by setting AT1_LAKEHOUSE.
Storage, not compute. The engine scans the same Parquet it always did; the saving is on what you store.
Snowflake needs S3, not R2. R2 hits a Snowflake region-signing defect where no single REGION value satisfies both its metadata and scan paths. The gateway path is unaffected.

Lakehouse cold tier — at1 lakehouse