Functions

Run SPQL functions, such as spql.str.upper or image decoding, on Arrow batches that your application already holds. You need no Project, credentials, or storage: give Functions a ScanDAL and evaluate calls on a batch. These are the same implementations that a Scan runs next to storage, so a value you compute locally, for example while preprocessing a training batch, matches the value a Scan would compute.

Functions::list includes each function's standard SQL aliases and their originating standard or system. SDK expressions use the canonical function ID; SQL frontends accept the aliases listed in the function reference.

Evaluate a batchCopy Anchor Link

Create Functions from the ScanDAL your client uses, embedded or remote, then evaluate named outputs:

use std::sync::Arc;

use spiral::arrow_array::RecordBatch;
use spiral::vortex::expr::col;
use spiral::{Error, Expression, Functions, NamedExpression, Scandal};

/// Compute camera labels and their character counts on a batch of demo frames.
pub async fn project_frame_metadata(
    scandal: Arc<dyn Scandal>,
    frames: RecordBatch,
) -> Result<Vec<RecordBatch>, Error> {
    let functions = Functions::new(scandal).await?;
    let camera = Expression::new(col("camera"))?;
    functions
        .evaluate(
            &[
                NamedExpression {
                    name: "frame_id".into(),
                    expression: Expression::new(col("frame_id"))?,
                },
                NamedExpression {
                    name: "camera_upper".into(),
                    expression: Expression::call("spql.str.upper", vec![camera.clone()])?,
                },
                NamedExpression {
                    name: "camera_characters".into(),
                    expression: Expression::call("spql.str.len_chars", vec![camera])?,
                },
            ],
            frames,
        )
        .await
}

The example returns frame IDs, uppercase camera labels, and character counts. evaluate computes all outputs together and returns batches that hold one row per input row, in order. Functions::new reads the function catalog once, so create it once and clone it.

Build calls with Expression::call and the function's full name. Pass parameters such as a format or a unit as literal expressions. Each argument must already have a type the function accepts, because SPQL never casts implicitly; add an explicit cast when the types differ.

Functions APICopy Anchor Link

project_stream checks the outputs before it reads any input, so a type error fails before the first batch, while an error in the data ends the output stream. The output pulls input on demand using its existing stream buffering. Dropping it stops queued work and further input polling; work that already started retains its inputs until it finishes. With remote ScanDAL, an error in the data can instead fail the call itself.

Inspect a planCopy Anchor Link

Inspect a projection before reading its input to see the selected implementations, dependencies, output types and required transfers:

use spiral::PlanDescription;

/// Print the common plan description without reading input batches.
pub async fn print_projection_plan(
    functions: &Functions,
    outputs: &[NamedExpression],
    input: &spiral::arrow_schema::Schema,
) -> Result<(), Error> {
    let schema = functions.bind(outputs, input).await?;
    if let Some(plan) = PlanDescription::from_schema(&schema)? {
        println!("{plan:#}");
    }
    Ok(())
}

PlanDescription reads the same versioned description from bind results, projected Scan schemas and evaluated output schemas. Its ordinary text format shows physical execution; {plan:#} also shows original operators and compiler decisions. The structured description is serializable as JSON. Literal values are omitted from both formats.

A bind result is a preview. Evaluated outputs and individual scan batches carry the actual plan and an execution ID; a host can bind again when execution starts. Compare the physical IDs to detect a selection change. Physical IDs belong to one execution host's lifetime, so they are not persistent cache keys across hosts or releases. A scan's attachment describes its projection, rather than the storage reader's complete pruning plan.

The description travels in reserved Arrow schema metadata, spiral.execution.plan, through existing operations. It does not change row types. PlanDescription::row_schema removes that attachment when comparing logical schemas. Older execution hosts can omit descriptions.

Profile an executionCopy Anchor Link

Enable profiling on the input stream to collect aggregate timing and buffer observations for the same physical nodes shown by explain:

use futures::{StreamExt as _, TryStreamExt as _};
use spiral::{BatchStream, ExecutionOptions};

/// Collect the executed plan and profile, including partial evidence on a batch error.
pub async fn analyze_projection(
    functions: &Functions,
    outputs: &[NamedExpression],
    input: BatchStream,
) -> Result<Vec<RecordBatch>, Error> {
    let input = input.with_execution_options(ExecutionOptions {
        profile: true,
        caller: Some("frame-preprocessing".into()),
        partition: None,
    })?;
    let mut output = functions.project_stream(outputs, input).await?;
    let diagnostics = output.diagnostics();
    let rows = output.by_ref().try_collect::<Vec<_>>().await;
    for plan in diagnostics.plans() {
        println!("{plan}");
    }
    rows
}

Keep the diagnostics handle to read the latest reports after an error or after dropping the stream. A successful report says completed; a dropped consumer marks its last observation with consumerCancelled. That observation can be older than the host's final state. Each execution has a separate ID, including concurrent executions of the same projection. Older hosts may return no profile.

Profiling is off by default. The report contains aggregate counters per node, with up to 1024 measured nodes, rather than a history of individual rows or batches. Each diagnostics source retains up to 64 execution reports and 16 MiB of their encoded descriptions; omitted() counts observations it could not retain. HTTP and native delivery preserve updates beside the batches, including empty summary batches, and the SDK removes that framing from your row columns. No telemetry collector is required.

Host elapsed time includes asynchronous waits. Blob I/O counters measure calls and bytes at the function's BlobIo boundary, before transport caching or coalescing; they are not object-store traffic. Input wait includes upstream work, and consumer wait measures when the runtime's adapter polls again, which can differ from when a remote application asks for another batch. Ordered wait measures completed batches waiting for earlier work or a later pull. These intervals can overlap and must not be added as a query wall-clock total.

Admission wait measures time waiting for a declared backend limit, such as decoder capacity, using the existing physical node IDs. Input buffer estimates and delivery reservations retain their named scopes; they do not measure total process memory. Generic memory admission and intermediate allocation reservations are not implemented. Device timing, device reservations, exact copied bytes and backend session reuse remain unknown when the backend supplies no observation. Inline storage projections report scan wait and output observations; their internal work is not timed separately.

For a reproducible report, save the structured plans alongside the host build, features, hardware, input batch shape and consumer behavior. Reports omit literal payloads, URLs and credentials. Caller correlation accepts only opaque IDs.

Independent asynchronous calls can overlap when their implementation explicitly permits it. Calls with effects or volatile results retain evaluation order within each batch, and errors preserve the earliest failing call in that order. Async calls inside conditional expressions fail during binding until masked execution is supported. The plan describes dependencies and scheduling choices using the same node IDs as the profile.

Blob payloadsCopy Anchor Link

These calls have no storage access of their own, so they read only blobs whose bytes are inline in the batch. A blob that points to storage fails. To read blobs stored with a Table, use project_stream_with_scans and pass the Scans that read those Tables; the SDK fetches fresh storage access for each one, and each blob must sit inside the storage of one of those Tables.

Use the functions in SQLCopy Anchor Link

To call the same functions in SQL, register them in a DataFusion session. See DataFusion.

AvailabilityCopy Anchor Link

ScanDAL runs scalar functions, which produce one value per row, on the CPU. Functions that run on a GPU or another device are planned. Hosted ScanDAL decodes H.264 video frames with spql.video.decode on its CPU.