Python SDK

The Python SDK lets a Python application, notebook or training job read exact Table versions as Arrow, write new versions and publish them as one commit, read blob payloads, and run SPQL functions on its own Arrow data. It is distributed as the spiral-sdk package and imported as spiral. Because it is built from the Rust SDK, Python code resolves the same resource addresses, opens the same Table versions and gets the same rows as Rust code, and it reads storage from your Python process with no extra service to run.

How it worksCopy Anchor Link

The package holds one native module compiled from the Rust SDK and embedded ScanDAL, so scans run inside your process. A client created with a ScanDAL endpoint runs them in the ScanDAL service instead. The choice is made when the client is created, from its arguments, so installing the package never decides where your code runs.

The SDK includes table execution and SPQL. SQL and the ADBC integration are distributed independently in spiral-sql, with their own native module.

Rows arrive as pyarrow record batches that share memory with the scan's output. Filters and computed outputs are expressions: columns, literals, comparisons, boolean logic and calls to spql.* functions. The Python builders produce the same encoded expression as the Rust and TypeScript SDKs, so one selection is the same value in every language.

Install spiral-sdk for Python access. The spiral CLI comes from spiral-cli, and the SQL terminal, local Connect server and ADBC driver come from spiral-sql. The SDK, CLI and SQL and Connect package are available on PyPI. Each package works independently and has its own version and releases, so you can upgrade a tool without upgrading the SDK. The SDK's cli and sql extras install the corresponding tools when an application needs them:

pip install 'spiral-sdk[cli,sql]'

The adbc extra is an alias for sql; harlequin installs spiral-sql[harlequin]. Neither the CLI nor SQL is included in a base SDK install. The extras accept tool versions below 1.0; the SDK and tool versions do not need to match.

The SDK, CLI and SQL packages include the Spiral Native Binary License for proprietary compiled runtime and CLI code. It restricts reverse engineering, decompilation and disassembly of that code. SDK wrappers remain MIT licensed, and third-party components retain their own licenses.

What the first version coversCopy Anchor Link

• A client configured from explicit arguments, with credentials from the SPIRAL_* environment variables or a saved spiral login.
• Tables opened at a resource address such as frames@main, or by exact identifiers.
• Scans with selected columns or computed outputs, a filter, a row limit and a batch size, read in stored order or one split at a time, and on request the counters of each read: the fragments that statistics pruned and the row windows the read visited. A scan can add each row's row ID as a column with row_id_column and keep only chosen row IDs with row_ids, and each split names its fragment in fragment_id.
• Writes that return a Table's next version: append, overwrite, delete_rows and writer, plus Project.write_table and Project.table_writer for the first version of a new Table. A writer uploads large blobs in parts, or packs many small ones together with pack_blobs, before the rows that reference them, and a writer of an existing Table also takes patch_rows and delete_rows for rows named by row ID in the ROW_ID_COLUMN column. A written version stays unpublished until a commit stages it.
• Row updates and merges: Table.patch_rows sets columns to expressions in the rows a filter matches, and Table.merge_rows merges Arrow rows by key columns, an upsert by default. Both record new values as patches beside the data, and rows keep their row IDs.
• Schema changes and new columns: Table.schema, and Table.alter_schema with add_column, drop_column, rename_column, set_nullable, widen_type and move_column, which write metadata only. Table.add_columns fills new columns from expressions, and Table.column_writer writes their values one fragment, or chosen rows, at a time.
• Change reads: Table.plan_changes reports which fragments and row IDs changed between two versions from metadata alone, Table.changes_since reads the changed rows as one scan with each row's change kind in the CHANGE_KIND_COLUMN column, and Snapshot.table_changes reads them for each commit that advanced a Table.
• Commits with the same shape as the Rust SDK's: project.branch("main") pins the branch head, head.table(address) opens Tables at it, and head.commit(message) stages update_table, create_table, rename_table and drop_table without any request. create_connection and drop_connection name and retire Connections on main. publish returns the branch's new head, so readers see every staged change or none.
• Publication failures as exceptions you can act on: BranchMoved and PublishConflict mean nothing was published, and PublishOutcomeUnknown carries the FrozenCommit to recover or publish again. Save a frozen commit as JSON with save and resume it with project.resume_commit after a restart; see Retries.
• Enrichments: head.commit().create_enrichment defines a value computed from each row of a Table, and head.enrichment(path).refresh() submits the Task that computes it and returns a Task to wait on.
• Ranged blob reads, and uploads in parts for Tables kept in a local directory, which tests and examples use without a Project.
• SPQL evaluation over Arrow batches that your application supplies. Given scans, an evaluation reads the blobs of the Tables those scans read, so a video decode over your own rows reads frames from an exact Table version.
• The SPQL data types as pyarrow extension types in the spiral.types module. Importing it registers every type, so columns read from scans, Arrow IPC or Parquet have their typed class, with one accessor per field, and each type has a constructor such as spiral.types.position_3d(x=..., y=..., z=...).
• The PyTorch integration, in the spiral.torch module.
• Every service's operations, such as client.projects, client.organization, client.authorization and client.tasks.

Manage resourcesCopy Anchor Link

Each service's operations are an attribute of the client, with the same names as the Rust and TypeScript SDKs. A method takes its path parameters in order, its query parameters as keyword arguments, and a request body as one typed value, and returns a typed response or a page iterator. Every request carries the client's current credential, renewed when it lapses.

import spiral
from spiral.services.project.wire import ProjectCreateProjectRequest

client = spiral.Client()
for project in client.projects.list_projects_items(page_size=100):
    print(project.id, project.description)
client.projects.create_project(ProjectCreateProjectRequest(description="Robot episodes"))

Each service's types are in spiral.services.<service>.wire and its errors in spiral.services.<service>.raw, for example spiral.services.tasks.raw.TasksConflictError. A type takes its fields as keyword arguments in snake case, with no default for a required field, and reads them back as typed attributes, including nested objects. A union takes exactly one of its variants, such as TasksTaskDefinition(table_maintenance=...); type names the variant it holds, which may be one added after your client was built. replace returns a copy with some fields changed, so a value read from one request can be sent back in another. A client from spiral.Client.local() has no control plane, so its resource requests fail.

AvailabilityCopy Anchor Link

For SQL from Python, the optional ADBC integration supplies DataFusion SQL through spiral_sql.adbc, with Harlequin available as another extra. The SDK itself does not include a SQL engine.

Recovering a write whose registration is unknown is planned for a later version. Until then, use the Rust SDK for that. The unrelated spiral package on PyPI also installs a top-level spiral module, so do not install both in one environment.