Docs · Integrations

TanStack Virtual + AT-1

Render a table with millions of rows in React from a single compressed .at1 file — no database, no API server, and only a fraction of the file ever read. AT-1 and TanStack Virtual line up almost perfectly.

Why they complement each other

TanStack Virtual:  renders only the rows currently on screen (a moving window).
AT-1:              stores the table compressed in row-groups and decodes only the
                   group a row lives in — reading a fraction of the file, no server.
Together:          the virtualizer's visible range -> the exact row-groups to decode.
                   A million-row table scrolls smoothly from one static .at1 file.

AT-1 stores a table as compressed row-groups, each with a zone-map and an integrity seal. That row-group is the natural unit of virtualization: the visible window touches one or two groups, so you decode one or two groups — not the file. The “compression” is table stakes; the point here is that the format is addressable, so a front-end can pull exactly the slice it's about to paint.

Install

npm install @tanstack/react-virtual
# plus the AT-1 query WASM (at1_block.js + at1_block.wasm — the build that powers /try).
# Serve them from /public/wasm and load at1_block.js once; it registers window.AT1Module.

1 · A 40-line wrapper over the AT-1 query WASM

Open the file, read its schema, and decode a single row-group on demand (cached). This is the same WASM that powers the /try demo.

// at1-rows.ts — a tiny wrapper over the AT-1 block-query WASM.
// Opens a queryable .at1, reads its schema, and decodes ONE row-group on demand
// (cached). Pure client-side: the file never leaves the browser.
let M: any;

export type At1Table = {
  h: number; ncols: number; nrg: number; total: number;
  types: string[]; rowsPerGroup: number[]; starts: number[];
};

export async function openAt1(url: string): Promise<At1Table> {
  M ??= await (window as any).AT1Module({ locateFile: (f: string) => `/wasm/${f}` });
  const bytes = new Uint8Array(await (await fetch(url)).arrayBuffer());
  M.FS.writeFile("/in.at1", bytes);
  const h = M.ccall("at1q_open", "number", ["string"], ["/in.at1"]);
  if (!h) throw new Error("Not a queryable .at1. Encode with: at1 compress qcolumnar " +
    "data.csv out.at1 --backend zstd --block-backend zstd --keep-queryable");
  const ncols = M.ccall("at1q_ncols", "number", ["number"], [h]);
  const nrg   = M.ccall("at1q_nrowgroups", "number", ["number"], [h]);
  const total = M.ccall("at1q_total_rows", "number", ["number"], [h]);
  const types = Array.from({ length: ncols }, (_, c) =>
    ["text", "int", "dec"][M.ccall("at1q_coltype", "number", ["number","number"], [h, c])]);
  const rowsPerGroup: number[] = [], starts: number[] = [];
  let acc = 0;
  for (let g = 0; g < nrg; g++) {
    const r = M.ccall("at1q_rows_in_group", "number", ["number","number"], [h, g]);
    starts.push(acc); rowsPerGroup.push(r); acc += r;
  }
  return { h, ncols, nrg, total, types, rowsPerGroup, starts };
}

// group decode cache: groupIndex -> { colIndex -> BigInt64Array }
const cache = new Map<number, Record<number, BigInt64Array>>();

export function getRow(t: At1Table, i: number): (bigint | null)[] {
  // which row-group owns global row i? (binary search the prefix sums)
  let lo = 0, hi = t.nrg - 1, g = 0;
  while (lo <= hi) { const m = (lo + hi) >> 1; if (t.starts[m] <= i) { g = m; lo = m + 1; } else hi = m - 1; }
  const local = i - t.starts[g];
  let cols = cache.get(g);
  if (!cols) {
    cols = {};
    const cap = t.rowsPerGroup[g];
    for (let c = 0; c < t.ncols; c++) {
      if (t.types[c] !== "int") continue;            // in-browser WASM decodes INTEGER columns
      const buf = M._malloc(8 * cap);
      const n = M.ccall("at1q_decode_int", "number",
        ["number","number","number","number","number"], [t.h, g, c, buf, cap]);
      const arr = new BigInt64Array(cap);
      for (let k = 0; k < n; k++) arr[k] = M.getValue(buf + 8 * k, "i64");
      M._free(buf);
      cols[c] = arr;
    }
    cache.set(g, cols);                              // decode each group at most once
  }
  return t.types.map((ty, c) => (ty === "int" ? cols![c][local] : null));
}

2 · Feed it to useVirtualizer

count is the full row total; the virtualizer only ever calls getRow for on-screen indices, so AT-1 only decodes the groups you can see.

// At1VirtualTable.tsx
"use client";
import { useEffect, useRef, useState } from "react";
import { useVirtualizer } from "@tanstack/react-virtual";
import { openAt1, getRow, type At1Table } from "./at1-rows";

export function At1VirtualTable({ url }: { url: string }) {
  const [t, setT] = useState<At1Table | null>(null);
  const parentRef = useRef<HTMLDivElement>(null);
  useEffect(() => { openAt1(url).then(setT); }, [url]);

  const rv = useVirtualizer({
    count: t?.total ?? 0,                 // could be millions
    getScrollElement: () => parentRef.current,
    estimateSize: () => 32,
    overscan: 16,
  });

  if (!t) return <div>Loading…</div>;
  return (
    <div ref={parentRef} style={{ height: 600, overflow: "auto" }}>
      <div style={{ height: rv.getTotalSize(), position: "relative" }}>
        {rv.getVirtualItems().map((vi) => {
          const row = getRow(t, vi.index);  // decodes only this row's group (then cached)
          return (
            <div key={vi.key}
              style={{ position: "absolute", top: 0, left: 0, width: "100%",
                       height: vi.size, transform: `translateY(${vi.start}px)`,
                       display: "grid", gridTemplateColumns: `repeat(${t.ncols}, 1fr)` }}>
              {row.map((cell, c) => <span key={c}>{cell?.toString() ?? "·"}</span>)}
            </div>
          );
        })}
      </div>
    </div>
  );
}

Any column type, or a remote file: window through SQL

For text-heavy tables or a .at1sitting in S3, skip client decode and let the virtualizer drive a windowed query — full rows, still only the touched blocks read. Pair this with TanStack Virtual's range / onChange.

// Any column types, or a remote file? Window through AT-1's SQL endpoint instead.
// TanStack Virtual gives you the visible range; fetch exactly that slice.
async function fetchWindow(startRow: number, endRow: number) {
  const r = await fetch("https://your-at1-cloud/sql", {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: "Bearer <token>" },
    body: JSON.stringify({
      sql: `SELECT * FROM data/table.csv WHERE id BETWEEN ${startRow} AND ${endRow}`,
    }),
  });
  return (await r.json()).rows;   // only the touched blocks are read server-side
}

What this buys you

• No backend to stand up. One static file on a CDN; the browser does the rest.
• Constant memory. Decoded groups are cached and bounded; the DOM holds only visible rows (TanStack Virtual) and RAM holds only visited groups (AT-1).
• Verified + addressable. Same file is byte-exact and integrity-sealed — the data you render is provably the data you stored.