Data Plane

Understand the part of Spiral that reads and writes your Tables in storage, where it runs, and what it needs from your application.

The data plane is the part of Spiral that touches your data. It reads and writes a Table's files directly in object storage and hands your code Arrow record batches. Its bounded operations work on exact Table versions, so scanning a very large Table needs no more memory than scanning a small one.

Connect is the collection of HTTP APIs exposed by a Connect host. SQL and ScanDAL are two of these APIs; a host advertises which it supports. Connect can run as a hosted service or on localhost through spiral connect. libspiral is Spiral's embedded native library, linked by the Rust SDK to run data operations inside your process. One API it implements is ScanDAL, the Scanning Data Access Layer: bounded scans and writes, blob transfer and SPQL evaluation. Connect exposes these operations over HTTP; embedding libspiral runs the same work inside your process, with the same results.

An example

When your application scans frames@main for the camera and frame_id columns of front-camera frames, the work divides like this. The SDK asks the Catalog which version frames@main means and asks Tables for storage access to it. The data plane then reads only the data for those two columns, drops the rows that are not from the front camera, and streams the rest to your code. No Table bytes pass through the control plane. Read rows shows the code.

What it does

Scans. It reads one Table version as a stream of Arrow batches. A scan can select columns, filter rows, compute new columns, and stop after a number of rows.
Writes. It creates, appends to and overwrites Tables by writing new immutable files. A write is not visible to readers until your application commits it; see Commits and Branches.
Expressions. Filters and computed columns are SPQL expressions that look at one row at a time. Because no expression needs other rows, memory stays flat.
Top-k and aggregates. A scan can return the rows that sort first, or counts, sums, minimums, maximums and histograms per group, in place of its rows. Each part of the scan returns a small partial result in one pass, and the SDK merges them; see Top-k and aggregates.

Vector search in the data plane is planned.

What it leaves to others

ScanDAL never looks up names, makes authorization decisions or publishes commits. Those are control-plane decisions, so the application that calls the data plane makes them first and passes in the results: an exact Table version, storage access for that Table, and the scan or write to run. Today the Rust SDK does this for you. It resolves frames@main to a version, gets storage access and renews it before it expires, and commits your writes. The Architecture page explains why the work is split this way.

SQL runs above ScanDAL in an application query engine. Joins, sorts of whole Tables and aggregates with more groups than a scan keeps use that engine's memory policy. Local Connect exposes DataFusion SQL to paired browsers through the DataFusion integration above the SDK. libspiral has no SQL engine, and hosted Connect serves no SQL. See Query engines.

The local SQL API streams complete query results. Applications limit results in their SQL; Table previews use ScanDAL. The TypeScript SDK exposes the schema and incremental result batches.

What it needs to reach

To read Spiral-managed storage, the data plane asks SPFS, Spiral's storage signing service, to sign each object request, and then sends the request to the bucket directly. SPFS sees only the request, never the bytes, but it must be reachable, so reads of managed storage cannot run fully offline. Reads of local files need neither SPFS nor the control plane. See Storage access.

Where it runs

Placement	Who runs it	Status
Inside your process	Your application, through embedded ScanDAL in the Rust SDK	Available through `spiral-sdk` on crates.io.
Spiral-hosted, per region	Spiral, as the ScanDAL service, for browser apps and SDKs without the native library	In development in us-east4, at `https://scandal.spiraldb.dev`. Other regions are planned.
In your own cloud	You, as containers on your network, reading storage with your own cloud identity	Planned

Today, select embedded ScanDAL in the Rust SDK, or pass https://scandal.spiraldb.dev to the SDK as its ScanDAL endpoint to use the hosted service; the SDK never selects the hosted service on its own. See Choose a backend. Every placement runs the same scan and write code, so moving between them will not change what a read returns or how commits behave.

A hosted ScanDAL service runs work only for a caller it can identify, because Spiral meters that work and must know whom it is for before it starts. Each request carries a short-lived token for ScanDAL, which the caller obtains by exchanging its own Spiral access token. ScanDAL checks the token's signature, audience and expiry with the authorization server's published keys, so it calls no other service to admit a request. The token grants no data: each request still reaches storage only through its own scoped storage credentials.

Desktop Studio embeds libspiral. The web app reads through Connect at the hosted ScanDAL service or a paired localhost endpoint; see Studio.