Read Rows

Read frames into your application as Arrow batches, or query it with SQL in DataFusion. Either way the SDK reads the Table's files directly from storage, and only the columns and rows that you ask for reach your process. Every scan reads one exact Table version, so the same request returns the same rows.

Before you startCopy Anchor Link

• A Table frames in the Project demo.
• A client from the Python, Rust or TypeScript SDK.

Scan a TableCopy Anchor Link

Open a Table session on a branch, resolve the Table, and read its batches. A Table session pins one Project commit for every Table that it resolves:

frames = client.project("demo").branch("main").table("frames@main")
print(f"resolved address: {frames.address}")
scan = frames.scan(limit=100)
rows = 0
for batch in scan.to_reader():
    rows += batch.num_rows

The scan reads one exact version. A later write on the branch does not change it. Open a new Table session, or call TableSession::refresh, to read newer state. resolved_address() reports the exact path@v<version> that the scan read. Record it with a result to make the read repeatable.

Dropping the batch stream stops the scan.

Select and compute columnsCopy Anchor Link

Set ScanOptions::projection to name the output columns and to compute values before Arrow delivery:

frames = client.project("demo").branch("main").table("frames@main")
scan = frames.scan(
    projection={
        "id": column("frame_id"),
        "is_front": eq(column("camera"), literal("front")),
    },
    limit=100,
)
print(f"output schema: {scan.schema}")
rows = 0
for batch in scan.to_reader():
    rows += batch.num_rows

Expressions read the Table's columns. They support columns, literals, casts, arithmetic, comparisons, boolean logic, null checks, and SPQL functions. Operand types must match; add an explicit cast when they differ. Set ScanOptions::filter with the same expression form to remove rows during the scan. An unsupported expression fails when the scan opens, before any rows stream.

Use either columns or projection, not both.

Read in parallelCopy Anchor Link

Set partitioned and max_streams to read one scan through several concurrent streams. max_streams caps the number of streams, whatever the size of the Table:

frames = client.project("demo").branch("main").table("frames")
scan = frames.scan(["frame_id", "camera"])
shares = [scan.stripe_splits(spiral.Shard(worker=worker, workers=4)) for worker in range(4)]

def count(share: list[spiral.WorkItem]) -> int:
    return sum(batch.num_rows for batch, _item, _next_row in scan.read_work(share))

with ThreadPoolExecutor(max_workers=4) as pool:
    rows = sum(pool.map(count, shares))

In TypeScript, lanes caps the splits that one scan reads at once. In Python, give each thread a share of the splits from stripe_splits and read it with read_work.

Partitioned streams return rows in no fixed order. A scan with a limit uses one stream in stored order, so the limit applies to the whole scan.

Query with SQL in your engineCopy Anchor Link

Register a Table session as a DataFusion catalog. DataFusion then plans and runs the SQL in your process, and the SDK reads the Tables:

use std::error::Error;

use datafusion::arrow::util::pretty::print_batches;
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::{SessionConfig, SessionContext};
use futures::TryStreamExt as _;
use spiral::Client;
use spiral_datafusion::{SpiralCatalogProvider, with_spiral};

/// Query the frames table resolved from a pinned snapshot of the demo Project.
pub async fn query_frames(client: &Client) -> Result<(), Box<dyn Error>> {
    let tables = client.project("demo")?.table_session("main").await?;
    let catalog = SpiralCatalogProvider::new(tables);
    let context = catalog.session(SessionConfig::new());
    let mut batches = context
        .sql("SELECT frame_id, camera = 'front' AS is_front, UPPER(camera) AS camera FROM \"frames@main\" LIMIT 100")
        .await?
        .execute_stream()
        .await?;
    while let Some(batch) = batches.try_next().await? {
        println!("{} frames with is_front flags", batch.num_rows());
    }
    Ok(())
}

Table names in SQL are Table addresses, so "frames@main" and "frames@c<commit>" both work. Supported projections and filters move into the SDK scan. The rest of the query runs in DataFusion. The catalog is read-only. See the DataFusion integration for the supported SQL features.

Verify the resultCopy Anchor Link

Compare the row count with the number of rows that you wrote. For a pinned read, run the scan twice with the same path@v<version> address. Both scans return the same rows.

TroubleshootingCopy Anchor Link

The SDK reads Tables only. Reading the files of a Location through the SDK is planned.