Storage Access

Your application receives storage access: short-lived permission to reach one resource's files. When your code reads frames, the Tables service checks your permission and grants access to that Table's files. The access expires, and the SDK renews it while your scan runs. The token lifetimes depend on the access method.

An exampleCopy Anchor Link

Your application opens frames@main with the Rust SDK. Behind the scenes, the SDK asks the Tables service for storage access to that Table and receives something like this:

{
  "rootUri": "spfsu://3f6c2a9e-5b1d-4c8e-9a7f-2e4b6d8c0a13/",
  "mode": "readOnly",
  "expiresAt": 1791910800,
  "access": { "type": "signed", "endpoint": "https://spfs.spiraldb.dev", "token": "…" }
}

The rootUri names the Table's directory without revealing the bucket, account or credential behind it. The data plane then asks SPFS, at endpoint, to sign each object request under that root, and sends the signed request to the bucket itself. The bytes go directly between storage and your process.

Managed and external storageCopy Anchor Link

Spiral reads two kinds of storage:

• Managed storage belongs to a Project. Tables, Enrichments and Vector Indexes keep their immutable files there. Each Project has one storage location, which can be Spiral-managed storage or a bucket in your own cloud account; see Storage.
• External storage stays under the control of you or a data provider. A Location, such as recordings over s3://example-robotics/recordings/, gives an existing folder a name in your Project, and a Connection holds the credential that reaches it.

Use managed storage for data whose versions and retention Spiral should manage. Use external storage when another system owns the files and Spiral should only read them.

Names you will seeCopy Anchor Link

The same idea appears under a few names in the API and in error messages:

What a binding holdsCopy Anchor Link

Ask the service that owns the resource for a binding:

To renew a binding, ask again before expiresAt. The Rust SDK gets and renews bindings for you. These calls belong to the SDK's host code; execution receives exact versions and scoped access and makes no control-plane calls. An application that uses the SDK does not need to call mount-management APIs. Direct HTTP integrations can use the resource access routes above.

How access is issuedCopy Anchor Link

Three checks precede signed storage access, so that no single service can hand out access on its own:

1. Your permission on the resource is checked. For a read, the Storage service checks the read permission that the owning service registered for the resource: table:read for a Table or an Enrichment, vector-index:read for a Vector Index, or location:read for a Location. For a write, the owning service checks it.
2. The Storage service checks that the resource's mount belongs to that service and that resource, and that the requested mode is within the mount's mode.
3. The Storage service requests the storage capability from the authorization server as its own client. The capability lists each directory it covers and the actions allowed there (RFC 9396 authorization details), names you for audit, and ends when your access token does. The authorization server checks each detail against its published schema, and issues storage capabilities to the Storage service alone.

The capability lasts at most fifteen minutes and ends when your access token expires. If a step fails, the request for a binding fails. Spiral does not substitute another access method after a failure.

How signing worksCopy Anchor Link

SPFS is a signing service, not a proxy. For each object request, it checks that the path lies inside the root the capability covers and that the action is allowed. It reads nothing else from the capability. Then it signs the request with the bucket credential that Spiral holds and returns a URL valid for at most five minutes, and never past the expiry of the capability. Your code sends that request to the bucket directly.

An address in a result tells you who reads the bytes:

Reading blob bytesCopy Anchor Link

A blob value holds either its bytes inline or a URI with a byte range and metadata. Creating, storing, comparing and selecting blobs do no I/O. Only two SPQL functions transfer data, and you call them explicitly:

• spql.io.fetch(blob) returns the payload as an inline blob; spql.blob.bytes gives its bytes.
• spql.io.stat(blob) returns object metadata without the payload.

So a scan can filter rows on metadata first and fetch bytes last, only for the rows that remain. In a Table scan, these functions read inline blobs and blobs stored in that Table, using the same storage access as the rows.

Reading a spiral:// address of another resource requires the execution host to configure URI resolution. The host checks access to that resource and supplies its storage binding. It can also configure access to provider URIs such as s3:// or gs://. Unconfigured addresses fail with an error.

Resolved storage is reused for other objects under the same authorized prefix until its credentials need renewal. Storage clients and connection pools are shared across reads, including after credentials are renewed.

Hosted execution, the native Python SDK and the Rust SDK's embedded backend resolve spiral:// addresses using your credentials and the named resource's read permission. Desktop Studio resolves them as the account signed in to each window, and Spiral Connect's SQL resolves them as the query's caller. Reads that hosted Studio sends through Spiral Connect use only the storage access each request carries. If a resolution is refused, for example because you lack the named resource's read permission, the read fails with that refusal. See the blob functions.

Writing blobsCopy Anchor Link

A write stores a blob value's inline bytes in the Table's own storage, as a range of a pack, spiral://<table-id>/blobs/<pack-id>, with its size and BLAKE3 digest. A payload of 4 MiB or more gets an object of its own, and equal payloads in one write share one range. A value that holds an address, of another Table's blob, a Location's file or a direct provider URI, is kept as a reference.

Nothing copies a referenced blob for you. To store its bytes in the Table, write spql.io.fetch(blob) instead: it holds the bytes inline, so the write packs them like any other inline payload. A reference to another resource fails to read once that resource is dropped. Construct a reference with spql.blob.external(uri).

To pack payloads before writing the rows that carry them, use ScanDAL's pack_blobs, or a blob pack in the Rust SDK; see Pack many small blobs.

Exact data in external storageCopy Anchor Link

A Project commit fixes which version of each managed resource a read selects. It cannot fix files in external storage that someone else can change. An object version, generation, ETag or content hash identifies the exact bytes; without one, the same URI can return different data later. When a training run or evaluation must be reproducible, keep that identity with the URI, or copy the data into a Table.

If a read stops after delivering some bytes, Spiral reopens the remaining range using the same object version or ETag. A transient failure while reopening uses the remaining retry budget for that read. A changed version fails the read, so bytes from different versions are never combined. A read without a version or ETag resumes only when the storage provider guarantees that the object key is immutable.

Location blob links are floating: spiral://<location-id>/<path> uses the Location's current source root, even when the link was saved in an older Table version. Object pins constrain the file at that root; they do not select an older Location configuration. Fetch the file into a Table when its bytes must travel with that Table's history.

If you bring your own bucketCopy Anchor Link

When a Project stores data in your bucket, a few settings are yours to manage. New Projects take their storage from the Organization default; see Store Tables on a Volume.

Large writes to S3-compatible storage use multipart upload. When a multipart write fails, the client tries to abort it, but a crash or a network failure can still leave parts behind. Add a bucket lifecycle rule that aborts incomplete multipart uploads. The rule removes only parts of uploads that never completed, never finished objects. Start with seven days, or a shorter interval that still exceeds your longest upload and retry window. Spiral never creates or changes lifecycle rules in your bucket.

Amazon S3 keeps incomplete multipart uploads forever by default, so add an AbortIncompleteMultipartUpload rule. If Spiral uses a prefix in a shared bucket, scope the rule to that prefix. In the console, open the bucket, select Management, create a lifecycle rule, select Delete incomplete multipart uploads, and enter 7 days. With the CLI or API, merge the rule into your existing rules first, because the put operation replaces the whole lifecycle configuration:

{
  "Rules": [
    {
      "ID": "spiral-abort-incomplete-multipart-uploads",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "<your-spiral-prefix>/"
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket <your-bucket-name> \
  --lifecycle-configuration file://lifecycle.json

AWS documents the lifecycle setup in detail.

Cloudflare R2 buckets include a default rule that aborts incomplete multipart uploads after seven days. To restore or change it, open the bucket's Settings in the dashboard and edit Object Lifecycle Rules. See the R2 object lifecycle documentation.

AvailabilityCopy Anchor Link

Direct browser access through SPFS is in development. The origins your Application registered restrict its storage capability. SPFS answers preflights without credentials and checks the origin before any signature. Object storage also needs CORS access for the Application's origin and exposed range headers. The hosted SPFS deployment does not yet support these preflights.

SPFS can also hand out a read-only provider credential narrowed to one directory, so that a client reads the bucket directly with GET and HEAD requests and no per-request signature. This is not enabled on the hosted service. It requires an unscoped credential, and storage decides from its configuration which storage it vends for. Scoped users and installations use signed requests, whose lifetime stays within the storage capability's expiry; they cannot obtain independent provider credentials. An app's own token cannot obtain provider credentials either.