Appendable tables

The qcolumnar container is a read-only archive. A real table grows. at1_table.AppendableTable adds append-only growth without rewritingwhat's already there: each append writes a new immutable, independently-verified segment (a queryable .at1) and records it in a manifest. The result is a growing, searchable, byte-exact record — the table-format play (vs Delta/Iceberg) for append-heavy data: logs, events, ticks, and audit trails.

Append a batch

from at1_table import AppendableTable

t = AppendableTable("trades_dir")
t.append_csv("batch1.csv")          # -> immutable segment seg_000000.at1
t.append_csv("batch2.csv")          # -> immutable segment seg_000001.at1

rows, stats = t.scan(where={"c0": (lo, hi)}, select=["c0", "c1"])
# stats: segments_total / segments_skipped / segments_scanned + block-level I/O

Every append_csv() compresses the batch with --keep-queryable(so a batch that the RAW fallback would win on size can never become an unqueryable segment and silently break the table's scan), records the segment's row count, byte size and sha256, and links a new event into the audit chain. Existing segments are never touched.

python at1_table.py trades_dir append batch1.csv
python at1_table.py trades_dir scan c0:1704067200000:1704067210000

Queries span all segments

scan() runs across every segment through an AT1Catalog with two levels of skipping:

  • File-level— the manifest's per-segment min/max rules out whole segment files before they're opened.
  • Block-level— each surviving segment's zone-map footer skips the row-groups it can rule out.

So a selective query over a growing table reads a tiny fraction of it. scan() returns stats with segments_total, segments_skipped and segments_scanned, and every segment stays byte-exact recoverable.

Compaction

Many small appends fragment a table. compact() merges segments into one new segment: it decodes each victim, concatenates the reconstructions (a newline is inserted between batches when one lacks a trailing newline, so rows never fuse), verifies the merged row count against the manifest, then verifies the new segment byte-for-byte against its own decode before the manifest is switched. Old files are reclaimed only after the new manifest is durably on disk.

t.compact()           # merge ALL segments into one
t.compact(upto=10)    # merge the first 10 segments, keep the rest

Hash-chained audit log

Every operation (append, compact) appends an event to a tamper-evident chain. Each link hashes the previous link, the op, the detail and the timestamp:

chain_n = sha256(chain_(n-1) || op || detail || ts)

Because the timestamp is part of the hashed material, history's whencan't be rewritten without breaking a link — and rewriting or reordering any past event breaks every later link. verify_chain() recomputes the whole chain from genesis and returns true only if history is untampered. verify_immutable() independently re-hashes every segment file to confirm appends never rewrote them.

t.verify_chain()      # True iff history recomputes end-to-end
t.verify_immutable()  # True iff every segment still matches its recorded sha256

Next

The same hash-chained ledger powers time-travel queries — reconstruct the table as it was after any past event. To tail a live stream into an appendable table, see streaming & live ingest.