Worker Pools
Provide the compute that runs Tasks: Spiral-managed Workers, or Workers on your own infrastructure.
Every Task runs on a Worker: a process with some CPU, memory, and maybe GPUs that picks up Tasks for one Project. The quickest way to get Workers is to turn on Spiral-managed compute, which starts Workers when Tasks are waiting and shuts them down when there is nothing to do. If your Tasks need GPUs, a private network, or to run next to your data, run Workers on your own machines instead, and group them into a Worker Pool.
For example, the demo Project turns on Spiral-managed Workers for Table
maintenance, so compaction of the frames Table needs no infrastructure at
all. It also runs a Worker Pool called gpu on the team's own GPU machines,
which sit next to the robot recordings.
Two ways to get compute
| Spiral-managed Workers | Your own Workers | |
|---|---|---|
| Setup | Turn on a provider, in Studio or with one request. | Run spiral-host worker start on machines you provide, with a workload to sign in as. |
| Scaling | Spiral starts and stops Workers with demand, down to zero. | You decide how many Workers run. |
| Hardware | CPU profiles from cpu-1 to cpu-16. | Anything you have, including GPUs. |
| Updates | Spiral updates Workers to each release. | You update them yourself. |
| Choose it when | You want Tasks to run with nothing to operate. | Tasks need GPUs, a private network, or compute next to your data. |
Both kinds of Worker take Tasks from the same queue, so you can use both in one Project. A Task does not say which kind runs it.
Workers, Pools, and sessions
| Part | Purpose |
|---|---|
| Worker | A process that runs Tasks for one Project. It reports its platform, its resources, and the Task specifications it can run. Start one with spiral-host worker start, or run a single Task with spiral-host worker run. |
| Worker Pool | A named group of Workers in one Project. It declares one platform, one resource size, the Task specifications its Workers run, and a limit on how many Workers it can have. |
| Worker session | One run of one Worker, from when it registers until it exits. Studio and the API show resource usage per session. |
| Managed provider | A Spiral-operated compute option, such as GKE US East. Turning it on creates Spiral-managed pools in the Project. |
A Worker does not have to be in a pool. A standalone Worker claims any Task in its Project that it can run. A Worker in a pool claims the Tasks its pool has reserved: a pool sets aside ready Tasks for its Workers, so two pools do not both start Workers for the same Task.
How a Task finds a Worker
You do not pick a Worker or a pool when you submit a Task. A Worker claims a Task only when all of these are true:
- The Worker and the Task belong to the same Project. The same check applies every time the Worker renews its claim or reports a result.
- The Worker runs the exact specification version of the Task.
- The Worker's resources cover the Task's resource requirements.
When several managed pools could run a Task, Spiral picks the one with the
smallest resource size that fits, so a cpu-2 Task does not occupy a
cpu-8 Worker. Pools with the same resource size can compete for the same
Task.
Spiral-managed capacity
Each managed provider creates one pool for each resource profile it supports:
| Provider | Profiles |
|---|---|
| GKE US East, GKE EU West | cpu-2, cpu-8, cpu-16 |
| Modal US East | cpu-1, cpu-2, cpu-4, cpu-8, cpu-16 |
Managed pools scale to zero: a pool with no Tasks waiting starts no Workers. A GKE Worker runs Tasks one after another and exits after 60 idle seconds. A Modal Worker runs one Task and exits.
Managed Workers sign in as a workload that Spiral creates in the Project with
the editor role the first time you turn on a provider. See
Turn on managed Workers.
Limits
Each pool has a maxWorkers limit on how many Workers can run at once. The
default is 1000. The limit is per pool; there is no shared limit for a
Project or a provider. Lowering the limit releases the Tasks the pool
reserved beyond it, so other Workers can take them, but Workers already
running an Attempt finish it.
Updates
Spiral updates managed Workers to each new release. An old Worker stops claiming new Tasks, finishes its current Attempt, and exits, and Spiral starts replacements while Tasks are still waiting. Managed pools keep their IDs, resources, and limits across releases. Spiral does not update Workers that you run yourself.
Availability
Worker Pools are in development. Client support lists the clients that can manage them.
Where it runs
Spiral hosts the Worker Pool API at
https://api.spiraldb.dev/api/tasks/projects/{project_id}/worker-pools and the managed
Workers API at https://api.spiraldb.dev/api/workers/projects/{project_id}/managed-workers.
Workers run on Spiral-managed compute or on your machines, and read and write
Table data directly in storage. See
Architecture.