Architecture
See what Spiral runs, what your application runs, and how one Table read and write moves between them.
Spiral separates permission and version decisions from data operations.
The control plane identifies callers, checks permissions, and resolves frames@main to one version.
The data plane reads and writes Table files in object storage.
It runs inside your application or through the hosted ScanDAL service.
Hosted execution receives Table bytes.
This page follows an embedded read and write through both planes.
The two halves
The control plane is the set of services that Spiral hosts: Authentication, Authorization, Organizations, Projects, the Catalog, and the services that own each kind of resource, such as Tables. Together they hold identities, permissions, names, version history and storage settings. They handle small requests, so their load stays the same whether you scan a megabyte or a petabyte.
The data plane reads and writes Table data. It works from exact inputs that the control plane has already decided: a resource ID, a version ID, and short-lived storage access for that one resource. Because it never looks up a name or asks for a permission decision itself, it needs no control-plane calls during execution. For embedded execution, the Rust SDK links the native library into your process. For hosted execution, the ScanDAL service runs the data plane. The host resolves resources and obtains storage access before execution.
One read and one write, step by step
The diagram follows one embedded read of frames@main and one write. The Rust SDK
talks to the control plane. The data-plane runtime, the native library, talks
only to storage.
- Get a token. Authentication exchanges
a credential for a short-lived access token. People sign in through a
browser, for example with
spiral login, and workloads use their own credentials. The SDK sends the token with each control-plane request. - Resolve the name. The SDK pins the head of the
mainbranch to one commit, and the Catalog tells it which resource and which version the pathframesnames at that commit. Names and branches stop here, so everything after this step works on exact IDs. See Resources for the address syntax. - Get storage access. The SDK asks Tables for access to that version. Tables asks Authorization whether the caller may read the Table and, if so, returns access to that Table's files only. The access expires within minutes, and the SDK renews it while a scan runs.
- Scan. The SDK hands the IDs and the storage access to the runtime. For each object it reads, the runtime asks SPFS, Spiral's storage signing service, to sign the request, and then sends the signed request to the bucket itself. The bytes go directly from the bucket to your process, and the runtime returns Arrow record batches to your code.
- Commit a write. For a write, the runtime writes new immutable files and reports the version it staged. The SDK registers that version with Tables and then asks the Catalog to commit it on the branch. Readers see the new version only once the commit succeeds. See Commits and Branches.
Where each kind of work runs
| Work | Runs in |
|---|---|
| Sign-in and access tokens | Authentication, in the control plane |
| Permission decisions | Authorization, in the control plane |
| Names, selected versions, and commit history | Catalog, in the control plane |
| Storage access for a resource | The service that owns the resource, such as Tables |
| Table scans, writes, and SPQL expressions | The data plane, embedded in your process or through hosted ScanDAL |
| Signing object requests | SPFS, which signs requests but never receives the bytes |
| SQL, joins, and aggregation | Your application's query engine, for example through the DataFusion integration |
Spiral's services do not run SQL for you. The data plane runs bounded scans that select columns, filter rows and compute expressions, and your application's own engine runs joins, sorts and aggregation. That keeps memory-hungry work in the engine that already knows how to manage it. See Query engines.
Why it is split this way
Execution determines the data path. Embedded execution transfers Table bytes directly between your process and object storage. Hosted execution transfers them through the ScanDAL service. The control plane receives metadata rather than Table bytes. See Security.
Reads are exact. The data plane receives version IDs, not names. Two readers with the same version ID read the same rows, even after a later commit moves the branch, which is what makes a job reproducible.
Access is narrow. Each grant of storage access covers one resource's files and expires within minutes. Code that holds it cannot use it to read any other resource. See Security.
What the split costs
The split is not free, and three costs are worth knowing:
- The SDK must reach the control plane to resolve names and to renew storage access. A long scan therefore still depends on the control plane, even though the runtime never calls it.
- Reads of Spiral-managed storage need SPFS to sign them, so they fail if SPFS is unreachable. Reads of local files need no SPFS.
- A write becomes visible only after a separate commit. If the commit response is lost, you do not know whether it happened. Recover the outcome instead of writing the data again; see Retries.