Query Engines

Spiral does not ask you to adopt a new SQL engine. You query Tables from the engine your application already uses; today that is Apache DataFusion. Spiral resolves an address such as frames@main to an exact version and streams it to your engine as Arrow, applying the column selection and simple filters your engine pushes down. Joins, sorts and aggregation stay in your engine.

An exampleCopy Anchor Link

This Rust program opens a DataFusion session over the demo Project and queries frames@main by name, without registering the Table first:

use std::error::Error;

use datafusion::arrow::util::pretty::print_batches;
use datafusion::execution::SessionStateBuilder;
use datafusion::prelude::{SessionConfig, SessionContext};
use futures::TryStreamExt as _;
use spiral::Client;
use spiral_datafusion::{SpiralCatalogProvider, with_spiral};

/// Query the frames table resolved from a pinned snapshot of the demo Project.
pub async fn query_frames(client: &Client) -> Result<(), Box<dyn Error>> {
    let tables = client.project("demo")?.table_session("main").await?;
    let catalog = SpiralCatalogProvider::new(tables);
    let context = catalog.session(SessionConfig::new());
    let mut batches = context
        .sql("SELECT frame_id, camera = 'front' AS is_front, UPPER(camera) AS camera FROM \"frames@main\" LIMIT 100")
        .await?
        .execute_stream()
        .await?;
    while let Some(batch) = batches.try_next().await? {
        println!("{} frames with is_front flags", batch.num_rows());
    }
    Ok(())
}

DataFusion sends the column selection, the camera = 'front' comparison and the limit into Spiral's scan, so the scan reads only the frame_id and camera columns and stops after 100 rows. It runs UPPER itself, because that is one of DataFusion's own functions. The DataFusion integration explains the setup.

Why SQL runs in your engineCopy Anchor Link

ScanDAL runs bounded scans and value operations through libspiral. A join, a sort without a limit, or a group-by over many keys needs memory that grows with the data. Your engine already manages that memory, and your application already chose its SQL dialect, its planner and its version, so Spiral leaves that work there instead of adding a hosted SQL service. The spiral-datafusion integration runs above the SDK; libspiral supplies scans and SPQL evaluation without containing a SQL engine. Local Connect can expose that integration over HTTP for paired Studio browsers.

An engine integration connects the two:

• The SDK resolves Table paths and pins exact versions, so every Table in a query reads one consistent state.
• The engine sends selected columns, supported filters and limits into each scan.
• The data plane reads storage and returns Arrow batches.
• The engine runs everything else.

Spiral's SPQL functions can also be registered in your engine, so you can call them in SQL over any input, not only over Tables.

An expression that your engine pushes into a scan gives the same result as it would in the engine. Where the two would differ, the engine keeps the work. For example, DataFusion's default integer arithmetic wraps on overflow while a scan fails instead, so DataFusion keeps that arithmetic.

Supported enginesCopy Anchor Link

SQL writes are not supported. Publish writes through an SDK commit.

Related pagesCopy Anchor Link

• DataFusion integration covers setup, quoting, pushdown and SPQL functions in detail.
• Expressions in scans explains what a scan can compute next to the data.