Read Changes

Keep something derived from frames up to date without rebuilding it: a search index, a cache in another system, or a model's predictions. Between two Table versions, Spiral reports which rows were inserted, updated and deleted, from version metadata, and reads only those rows. A change read therefore costs as much as the changed rows, not the whole Table, and needs no key column of yours: each row is named by its row ID, which stays the same when maintenance rewrites files.

Before you startCopy Anchor Link

• A Table frames in the Project demo, and the commit ID of an earlier state, such as the commit that the last run of your job read. See Select an exact state.
• A client, from the Python SDK or the Rust SDK.

Read row IDsCopy Anchor Link

A scan returns each row's row ID when you name a column for it, and reads only the rows whose row IDs you list:

main = client.project("demo").branch("main")
scan = main.table("frames").scan(
    ["frame_id", "camera"],
    row_id_column=ROW_ID_COLUMN,
    row_ids=[range(1_000, 2_000), range(5_000, 5_100)],
)
print(scan.schema)

row_id_column adds a last output column of that name, a non-null uint64. Name it ROW_ID_COLUMN (s:row_id), the column that row-ID writes take, so a scan's output can go straight back into a write. row_ids keeps only the rows in its half-open ranges: planning skips files whose row IDs cannot match, and the read enforces the ranges exactly. Both combine with columns, a projection and a filter.

Plan the changesCopy Anchor Link

plan_changes compares two versions of one Table from their metadata. It opens no data file, so it is cheap enough to run before deciding what to do:

project = client.project("demo")
current = project.branch("main").table("frames")
earlier = project.snapshot_at(since).table("frames")

plan = current.plan_changes(earlier)
for fragment in plan.fragments:
    match fragment.change:
        case RetainedFragment(patched, deleted):
            print(f"{fragment.fragment_id} kept: {len(patched)} patched, {len(deleted)} deleted ranges")
        case AddedFragment(appended, _, _):
            print(f"{fragment.fragment_id} added: {len(appended)} appended ranges")
print(f"{len(plan.removed_fragment_ids)} fragments removed")

The plan has one entry for each fragment of the later version, in stored order. Each entry is one of two kinds, with row IDs as sorted, disjoint, half-open ranges:

The plan also lists the fragments of the earlier version that the later one no longer has, and retyped_columns, the columns whose type differs between the versions. Pass columns to count only changes to those columns: a patch of camera does not mark a row patched for a plan of image.

The earlier version must be the same Table at this version or an earlier one.

Read the changed rowsCopy Anchor Link

changes_since reads the changed rows of both versions as one scan:

project = client.project("demo")
current = project.branch("main").table("frames")
earlier = project.snapshot_at(since).table("frames")

changes = current.changes_since(earlier, images="after", columns=["frame_id", "camera"]).to_table()
print(changes.group_by(CHANGE_KIND_COLUMN).aggregate([(ROW_ID_COLUMN, "count")]))

Every output row carries its row ID in ROW_ID_COLUMN and how it changed in CHANGE_KIND_COLUMN (s:change_kind): insert, delete, update_before or update_after. The columns, projection and filter apply to each image of a row. Choose the images with images:

A row that maintenance moved without changing it is not reported. Across an overwrite of the whole Table, every old row is deleted and every new row is inserted. With a filter, an update is reported when either its old or its new values match, so a row that stops matching still reaches you. Old values are read from the earlier version with the same columns, which must have the same names and types there.

The result is an ordinary scan: read it as batches or into one Arrow table. It has no splits to read as work items.

Read changes commit by commitCopy Anchor Link

To see each commit's change separately, as an audit log or a replication stream needs, read them from a branch head back to an earlier commit:

project = client.project("demo")
main = project.branch("main")
for commit in main.table_changes("frames", project.snapshot_at(since), images="keys"):
    print(f"commit {commit.commit_id}: {commit.changes.to_table().num_rows} changed rows")

table_changes follows the branch's first-parent commit history from the earlier commit, oldest first. Each commit that changed the Table contributes its commit ID, the Table versions before and after it, and a scan read as changes_since reads it, with the same options. The earlier commit must be on that history, and the path must name the same Table at both commits.

Verify the resultCopy Anchor Link

Run the read twice with the same two versions: it returns the same rows, because both versions are immutable. To check the counts, read the changes with keys: the earlier version's row count, plus the insert rows, minus the delete rows, equals the later version's row count.