Expressions in Scans

When your application scans a Table, it can attach a filter and a list of computed columns, and the data plane evaluates them next to the data. Only the rows and columns you asked for come back. These expressions are written with SPQL, Spiral's library of typed functions over values such as strings, timestamps and blobs. Each expression looks at one row at a time, which is why a scan's memory use stays flat however large the Table is.

An exampleCopy Anchor Link

This Rust scan of frames returns two columns for each frame: the frame ID, and a boolean that is true for the front camera.

let tables = client.project("demo")?.table_session("main").await?;
let table = tables.table("frames@main").await?;
let scan = table
    .scan(ScanOptions {
        projection: Some(vec![
            NamedExpression {
                name: "id".into(),
                expression: Expression::new(col("frame_id"))?,
            },
            NamedExpression {
                name: "is_front".into(),
                expression: Expression::new(eq(col("camera"), lit("front")))?,
            },
        ]),
        limit: Some(100),
        ..Default::default()
    })
    .await?;
println!("output schema: {}", scan.schema());
let mut batches = scan.into_stream();
let mut rows = 0;
while let Some(batch) = batches.next_batch().await {
    rows += batch?.num_rows();
}

The scan never reads the image or embedding columns, because no output uses them. It also reports its output schema before it reads any batch, so your code can check the shape of the result up front.

One row in, at most one row outCopy Anchor Link

A scan accepts two kinds of expression:

• A filter, a boolean expression that decides which rows to keep.
• Named outputs, a list of expressions that each have a unique name. The scan returns these outputs instead of the stored columns.

Both read the Table's stored columns. An output cannot refer to another output by name, and one scan uses either a plain column list or named outputs, not both. If you ask for no outputs at all, you get batches with no columns but the original row counts, which is a cheap way to count rows.

Every expression is row-local: one row goes in, at most one row comes out, and no expression reads another row. Work that needs many rows at once, such as joins, sorts and grouping, belongs in your application's query engine; see Query engines.

What expressions can doCopy Anchor Link

Scan expressions support:

• columns and literals;
• explicit casts;
• arithmetic, comparisons and boolean logic;
• null checks;
• structs built from named expressions, including nested structs;
• SPQL function calls, such as spql.str.upper.

There is no implicit type conversion, so both sides of an operation must have the same type. Add an explicit cast when they differ. Write literal function arguments, such as a format or a unit, as literal expressions.

Call an SPQL function by its full name, for example Expression::call("spql.str.len_chars", ...). The SPQL reference lists every function and the types it accepts.

Function support comes from the execution host's available functions. Filters must return boolean values and run synchronously; named outputs can include asynchronous functions, such as functions that read blob bytes.

When an expression failsCopy Anchor Link

The data plane checks each expression against the schema of the exact Table version when the scan opens:

• An unknown function, an unsupported feature or a type mismatch fails the scan before it reads any batch, so a mistake costs no I/O.
• An error that depends on the data, such as a failed cast or an integer overflow, fails the batch stream when it happens.

Using the same functions on your own batchesCopy Anchor Link

The same SPQL functions run on Arrow batches that your application already holds, with no Table and no storage access. This example computes upper-case camera labels and their lengths:

use std::sync::Arc;

use spiral::arrow_array::RecordBatch;
use spiral::vortex::expr::col;
use spiral::{Error, Expression, Functions, NamedExpression, Scandal};

/// Compute camera labels and their character counts on a batch of demo frames.
pub async fn project_frame_metadata(
    scandal: Arc<dyn Scandal>,
    frames: RecordBatch,
) -> Result<Vec<RecordBatch>, Error> {
    let functions = Functions::new(scandal).await?;
    let camera = Expression::new(col("camera"))?;
    functions
        .evaluate(
            &[
                NamedExpression {
                    name: "frame_id".into(),
                    expression: Expression::new(col("frame_id"))?,
                },
                NamedExpression {
                    name: "camera_upper".into(),
                    expression: Expression::call("spql.str.upper", vec![camera.clone()])?,
                },
                NamedExpression {
                    name: "camera_characters".into(),
                    expression: Expression::call("spql.str.len_chars", vec![camera])?,
                },
            ],
            frames,
        )
        .await
}

Outside a scan, functions that read blob bytes work only on blobs whose bytes are stored inline, because there is no storage access to fetch the rest. See SPQL functions in Rust.

Limits and wire formatCopy Anchor Link

Expressions are sent as protobuf bytes; an HTTP request carries the same bytes as base64 in JSON. The Rust SDK builds them for you. These limits apply before the data plane decodes an expression:

List functions apply a fixed operation to every element; they do not take a function of your own as an argument.