Storage

Every Table, Enrichment, and Vector Index in a Project stores its files in a bucket: Spiral storage to start, or your own bucket when the data must stay in your cloud account. Spiral gives each Resource its own folder in that bucket and hands out short-lived access to one folder at a time, so no client ever holds a key to the whole bucket.

Your application normally uses the Tables, Enrichments and other Resource APIs. Those services allocate storage and the SDK obtains access for reads and writes. You use Storage configuration when you choose where a Project stores new Tables, register your bucket, or administer a Project's existing file system; you do not allocate a folder for each Table yourself.

Storage is not the same as a Location. Project storage holds the data that Spiral manages. A Location names data that already exists elsewhere and only reads it.

An exampleCopy Anchor Link

The Project demo uses an S3 bucket in your AWS account, example-spiral, under the directory /demo/. When your application creates the Table frames, the Tables service creates a folder for it, and the files land there:

s3://example-spiral/demo/
  ├─ <frames folder>/        Table files and version metadata
  ├─ <is_blurry folder>/     Enrichment values
  └─ <frame-search folder>/  Vector Index files

When your application reads frames, it receives access to the frames folder only, for a few minutes, and reads the files directly from S3. See Storage access for how that access is issued and renewed.

VolumesCopy Anchor Link

A Volume is a bucket, or a prefix in one, whose contents Spiral manages: it creates a folder <project-id>/resources/<resource-id>/ there for each new Table, Enrichment and Vector Index, writes, compacts, and deletes its files, and gives nobody access beyond one folder. A Volume is either:

• Spiral storage: one of Spiral's buckets. Every Project can use it, with no credentials or setup.
• Your bucket: a bucket in your own cloud account that an Organization owner registers. Projects in that Organization can use it. Spiral still manages the Tables on it, so give it a bucket or a prefix that nothing else writes to. An Amazon S3 or Google Cloud Storage bucket trusts the Volume's own identity, as a Connection does, and Spiral stores no secret for it. An R2, MinIO, or RustFS bucket takes an access key. A bucket on a Ceph gateway takes a key or the Volume's identity. Ceph does not tell a write from an overwrite, so the credentials Spiral hands out for it can also replace objects. Every Volume must let Spiral create, read, list, and delete objects under the root. Support for other S3-compatible stores, such as Tigris and Hetzner, is planned.

Each Project stores its new Tables on one Volume. A Project administrator chooses it in the Project's settings; without a choice, a new Project uses Spiral storage from its Organization's default provider. Changing the Volume affects new Tables only: existing Tables keep their folders, and Spiral does not copy files. An Organization owner can retire a Volume, so that it takes no new Tables, or revoke it, so that Spiral grants no new access to the Tables on it.

To read data that you manage yourself, in place, use a Connection instead: Spiral never deletes through a Connection, and no Table lives on one. See Store Tables on a Volume.

File systemsCopy Anchor Link

Use app.spiraldb.com or the spiral CLI to manage volumes and file systems. The legacy file-system management endpoints are deprecated and continue to work during migration.

Before Volumes, the storage of a Project was its file system. A Project created before Volumes keeps storing new Tables and Vector Indexes on its file system until an administrator chooses a Volume. It has one of these types:

A builtin file system always uses the directory /<project_id>/ of its provider. The other types take a bucket and an optional directory, which starts and ends with /. An upstream file system lets several Projects share one bucket configuration; the other Project must use s3, gcs, s3like, or r2.

Spiral no longer accepts new gcs file systems. These file systems authenticate to storage with one shared Spiral identity. The provider's trust policy does not distinguish individual Projects. Spiral checks resource permissions before it issues storage access and limits signed requests to the permitted directory. Existing configurations remain supported.

To read your Google Cloud Storage bucket, create a gcs Connection. Configure your service account to trust that Connection alone.

A Project's file system cannot be changed. To store new Tables in another bucket, select a Volume for the Project; see Store Tables on a Volume.

MountsCopy Anchor Link

Each Resource's folder is recorded as a mount. Services create a mount when they create a Resource; you do not create mounts for managed data. A mount records the folder, whether it is read-only or read-write, and the Resource that owns it. Its opaque alias names it in older spfsu:// URIs and carries no bucket name or credential.

Blob values written into a Table, such as images, are stored in packs in the Table's own folder and addressed as spiral://<table-id>/blobs/<pack-id>. New values do not use spfsu://.

Remove a mount only when its Resource no longer needs it: a removed mount stops granting access to its folder.

A Table or Vector Index on a Volume has no mount. Reading it gets short-lived access to its folder at the bucket, where Spiral can narrow a credential to the folder, or signed access through Spiral otherwise; see Storage access.

Where nextCopy Anchor Link

• Store Tables on a Volume: register your bucket and choose where a Project stores new Tables.
• Inspect Project file systems: see a Project's file system and inspect or remove its mounts.
• Storage access: how a client gets short-lived access to one folder.

AvailabilityCopy Anchor Link

Storing Tables on Volumes is in development. A Project created before Volumes were turned on keeps storing new Tables on its file system until an administrator chooses a Volume. A Project can choose any of Spiral's Volumes, in any region; limiting an Organization's Projects to its permitted regions is planned. On your own Amazon S3 and Google Cloud Storage buckets, Spiral writes new files but does not yet delete the ones that Tables no longer need, so they stay in the bucket until compaction cleans them up in a later release.

Where it runsCopy Anchor Link

Spiral's control plane keeps the Volumes, each Project's choice, the file system configuration and the mounts, and issues access to one folder at a time. Object bytes move directly between your bucket and the application or Worker that reads or writes them. See Architecture.

The Storage HTTP reference documents every operation.