How Tables Are Stored

A Table never edits a file. Every write adds new immutable files to your Project's bucket and then publishes a small metadata object that says which files make up the new version. This one rule explains how Tables behave: a pinned version reads the same rows forever, an append costs as much as the new rows and no more, and maintenance can reorganize files while readers keep working.

This page explains the storage model at the level you need to reason about versions, cost, and the files in your bucket. You do not call any of it directly: the SDK reads and writes Tables for you.

What is in your bucketCopy Anchor Link

Take the Table frames, with the columns frame_id, camera, captured_at, image, and embedding. After a few appends, its directory in Project storage holds two kinds of object:

frames version v7 (one root object)
  ├─ schema
  ├─ fragment list, inline or in manifest files
  │    ├─ fragment A ─► files: frame_id, camera, captured_at, image,
  │    │                        and the shape of embedding
  │    │              ─► files: the values of embedding
  │    └─ fragment B ─► ...
  └─ small change records: row patches and deletion vectors

• Data files hold the column values. They use Vortex, an open columnar file format.
• Metadata objects describe one exact version: a root object for the version, plus manifest files that list the fragments when the Table grows large.

Each Table version is one root object. The version ID that you see in an address such as frames@v<version> is the ID of that object. To read a version, the SDK opens its root and follows the references in it. It does not list the bucket, look for the newest file, or replay older versions, so an old version opens as quickly as the current one.

Spiral's control plane does not hold the list of files in a version. It records which Project owns the Table and where its storage is, which version each commit selects, and the request IDs that make retries safe. Everything else about a version is in the objects in your bucket.

Fragments and column filesCopy Anchor Link

A Table stores its rows in fragments: large groups of rows, typically 250,000 to 1 million each. A fragment never changes after it is written. The writer fills a fragment in batches of rows. For each batch, one file holds the top-level columns, such as frame_id, camera, captured_at, and image. Each struct or list column adds files for its nested values, as Structs and lists explains. Maintenance later merges small files into larger ones of about 512 MiB.

This layout is why adding a column is cheap. Values for a new column, such as a sharpness score for every frame, go into new files that attach to each fragment beside the existing ones. The images and the timestamps are not rewritten.

It is also why a filtered read of a wide Table is fast. A scan reads the columns in the filter first, decides which rows match, and only then reads image or embedding, for the matching rows only.

Structs and listsCopy Anchor Link

A Table splits a struct or list column into its nested fields and stores each nested field as an ordinary column. A nested value therefore compresses, carries statistics, and loads like a top-level column of the same type, instead of being packed into one opaque value for each row.

Take a detections column on frames: a list of the objects found in each frame, where each object is a struct with a label and a score.

The writer stores these three rows at two levels, each with its own row count:

row level: one row per frame
  frame_id             f-000184    f-000185    f-000186
  detections (shape)   start 0     start 2     start 2
                       length 2    length 0    length 1

element level of detections: one row per detection
  label                person      forklift    person
  score                0.91        0.77        0.88

A list keeps its shape at the level that holds it: for each row, where its elements start, how many there are, and whether the list is null. The elements go in separate files, with one row for each element across the whole batch, so the three frames above produce three element rows. The shape is what ties an element to its frame, so the element files carry no frame ID or row ID. A fixed-size vector such as embedding is stored the same way: its values form one column of numbers in their own files, and a scan that does not select embedding never opens them.

A struct keeps only its nulls at the level that holds it. Its fields go in their own files at that same level, so they have the same row count: one row per frame for a struct column, or one row per detection for the structs inside detections. The rule repeats at every depth. A list nested in another list's elements adds a further element level, with its own shape and files.

Because each nested field has a stable field ID of its own, renaming score or adding a nullable field inside the struct changes metadata only. The layout also sets two limits:

• A scan reads a struct or list column whole. Selecting detections reads its shape and every nested field, even if the query uses only score. To read a nested field cheaply, store it as a top-level column.
• A row update replaces a whole list value. It cannot change one element in place, because an element has no row ID of its own.

Row IDsCopy Anchor Link

Every committed row has a hidden row ID: a 64-bit number that is unique within the Table and never reused. New rows get the next IDs in append order. When maintenance merges or reorders files, every row keeps its row ID. A deleted row's ID is retired, not handed to another row.

Row IDs let Spiral point at rows without a join on your columns. Each Vector Index entry records the row ID of its source row, so a search result leads back to the exact row even after maintenance moves the files.

A row ID is not a primary key, and a scan returns it only when you ask for it; see Read row IDs. Edits that change or delete chosen rows name them by row ID, and change reads report each changed row by it. Tables do not enforce primary-key or unique constraints. Keep your own identifier, such as frame_id, in an ordinary column. Duplicate values are allowed.

What each change writesCopy Anchor Link

Because files are immutable, each kind of change adds a small amount of new data instead of rewriting what is there:

Readers apply patches and deletion vectors while they scan, so a change is visible as soon as its commit lands. Over time these change records add up, and maintenance folds them back into the data files.

Row updates, such as patch_rows, write the small change record above: they are cheap to write, and reads merge them with the data until maintenance folds them in. Deletions are recorded as deletion vectors, because reading one costs little.

Versions and concurrent writersCopy Anchor Link

Every write publishes one new Table version through a Catalog commit. The commit names the version that the write started from. If another writer published first, the commit reports a conflict and publishes nothing, so no write is lost.

The format can reconcile some of these conflicts automatically. Two appends from the same version touch different rows, so the later one can be renumbered onto the new version and reuse the files that it already uploaded. Two changes to the same cell, or a delete and an update of the same row, always conflict. The Rust SDK does not reconcile yet: after a conflict, pin the branch again and repeat the write. See Write rows.

PruningCopy Anchor Link

A scan uses metadata to skip data that cannot match. The manifest entry of each file records statistics such as the minimum and maximum of its columns. Before a scan opens a file, it checks each part of an AND filter against them: a scan for captured_at after yesterday skips every file whose maximum captured_at is older, and plans no work for a fragment when none of its files can match. A column that a patch may have changed does not prune. Inside a file, Vortex keeps similar statistics for each chunk of rows, so a file that survives is still read selectively.

Each file records three statistics for each top-level column when it is written: minimum, maximum, and null count. The values come from the Vortex file footer, so the write does not scan the data again. Boolean, number, and decimal columns get all three. Timestamp, date, and time columns also get all three, stored as the underlying integer value. Floating-point columns also get a NaN count, because their minimum and maximum skip NaN values: a scan prunes by those bounds only in files that hold no NaN. A column's statistics are chosen when the column is created, so a floating-point column created before Tables recorded NaN counts has none and does not prune by its bounds.

String and binary columns get a null count and two bounds. Vortex keeps at most 64 bytes of a value in the footer, so a longer minimum becomes a shorter prefix and a longer maximum is rounded up. These bounds still contain every value of the file, so scans prune by them: camera = 'front', a range comparison, and a LIKE pattern that starts with a fixed prefix, such as 'front%', all skip files whose bounds rule out a match. The bounds are not the exact minimum and maximum, so MIN and MAX do not use them. Other extension types, such as UUID, get only a null count. Nested fields get no file statistics, though Vortex still keeps chunk statistics inside their files.

Query integrations also use these statistics to plan reads without opening data files. They distinguish exact values from conservative bounds and leave unavailable facts unknown. A partial read or a limit can make an exact value a bound; patches, deletions, missing statistics, or a changed physical type can make it unavailable.

Pruning works best when similar values are stored together. Appends keep their input order, so a Table that receives frames in time order prunes well on captured_at. The format can also record partition specs and sort orders, with the same transforms as Apache Iceberg, but Tables are written unpartitioned and unsorted today.

Schema changesCopy Anchor Link

Each column has a stable field ID, and files store values by field ID, not by name. Some schema changes therefore change metadata only and rewrite no files:

• rename or reorder a column or a nested field;
• add a nullable column, or a nullable field inside a struct, which reads as NULL for existing rows;
• drop a column;
• relax NOT NULL to nullable; and
• widen a number without loss, such as i32 to i64 or f32 to f64.

Older versions keep the schema that they were written with. Other changes, such as integer to float, need a new column. Change the schema shows how to make each change from the SDKs.

MaintenanceCopy Anchor Link

Many small appends leave many small files, and updates and deletes leave change records that every scan must apply. Maintenance rewrites them into fewer, larger files in the background. It publishes an ordinary new version and guarantees that:

• every row keeps its row ID and its values;
• the row count does not change;
• older versions still read exactly as before; and
• a write that lands while maintenance runs still applies on top of the rewritten files.

So maintenance changes cost and speed, never results. Run it as a Task, once or on a schedule. See Maintain a Table.

RetentionCopy Anchor Link

Every version that a commit selects stays readable, so a training run that pinned a commit can read the same rows later. Files that no version references any more, such as the inputs of a maintenance rewrite, also stay in your bucket today: physical deletion of unreferenced files is planned. Expect storage use to grow with history until it ships.