Run your own Workers

Run Workers on your own machines when Tasks need a GPU, a private network, or to sit next to your data. Your Workers take Tasks from the same queue as Spiral-managed ones. You supply the machines, a workload for the Workers to sign in as, and, if you want them, an autoscaler and a Worker Pool.

PrerequisitesCopy Anchor Link

• You have the Project editor role to create a Worker Pool.
• A workload with the Project editor role for the Workers. Tasks run with this workload's permissions, so it needs access to every Resource the Tasks read and write.
• The spiral-host execution host on each Worker machine.

Start a WorkerCopy Anchor Link

Set the workload's credentials, then start the Worker. This example uses a private key; on a platform with its own identity tokens, set SPIRAL_WORKLOAD_ID instead:

export SPIRAL_CLIENT_ID=work_01GHI
export SPIRAL_PRIVATE_KEY="$(cat work_01GHI.jwk)"
spiral-host worker start --project demo

The Worker registers its platform, its CPU and memory, and the Task specifications it can run, then claims and runs Tasks one after another. A Worker started this way is standalone: it claims any Task in the Project that it can run. See the spiral-host worker reference.

Run one Task on this machineCopy Anchor Link

To run a single Task without keeping a Worker running, give spiral-host worker run a Task definition, the same definition object that Submit Tasks sends. Save this one, which maintains the frames Table on the main branch, as maintain-frames.json:

{
  "type": "tableMaintenance",
  "tableMaintenance": {
    "readTarget": {
      "type": "latestOnBranch",
      "latestOnBranch": {"branchId": "0199c3a8-6a0e-7c41-9f2a-3a4c8f1e2b77"}
    },
    "tableResourceId": "0199c3a8-7b1f-7d02-8e5c-1f2d3c4b5a69"
  }
}

spiral-host worker run --project demo maintain-frames.json

The Worker submits the Task claimed by its submitter: only a Worker signed in as you can claim it, and no Worker Pool does. It then claims and runs the Task on this machine and prints its result: the versions it published. Pass - to read the definition from standard input. Because you chose the machine, only the Task's handler is checked against it, not its resource profile.

The Task is otherwise an ordinary Task: it appears in Studio, waits behind an overlapping Task, and retries within its attempt limit on this machine. If no Worker claims it within five minutes of it becoming ready, it fails with not_claimed.

spiral-host worker run --task <TASK_ID> runs a Task you submitted this way from elsewhere, such as an SDK, in that Task's own Project.

The Studio desktop app runs Tasks the same way: choose Run on this machine on a Location, Vector Index, or keyed table action.

Collect Worker diagnosticsCopy Anchor Link

Use the global --json-logs option to write JSON diagnostic logs to standard error. Set RUST_LOG to change the log filter; the default is info. Command results keep their normal standard output format.

Spiral-managed Kubernetes and Modal Workers use JSON logs and start with RUST_BACKTRACE=1. Each isolated Task executor also enables JSON logs and sets RUST_BACKTRACE=1 in its process environment.

Identify a Worker containerCopy Anchor Link

Each Worker session can record diagnostic tags as string key-value pairs. In Studio, open a Task Attempt, then select its Worker to inspect these values. Kubernetes Workers report Pod, namespace, and container identifiers, with GKE cluster details when available. Modal Workers report their container ID, region, cloud provider, and environment when available.

To attach tags to another Worker, set environment variables with the prefix SPIRAL_WORKER_TAG_. The Worker removes the prefix and converts the remaining key to lowercase. For example, SPIRAL_WORKER_TAG_MACHINE_ID records machine_id.

The tags remain available after the Worker session expires. They help identify the execution environment and do not grant access to logs. Use non-secret values because Project members can inspect them. Each session accepts up to 32 entries, with keys of 1 to 256 bytes and values of 1 to 1024 bytes. Keys and values must exclude ASCII control characters (0x00–0x1F and 0x7F). Invalid tags fail Worker registration with an error that identifies the field.

Group Workers in a PoolCopy Anchor Link

Create a pool to give a group of Workers a name, one resource size, and a limit on how many can run. The capabilities say what every Worker in the pool must support. This creates the gpu pool for the team's GPU machines:

curl -X POST https://api.spiraldb.dev/api/tasks/projects/demo/worker-pools \
  -H "Authorization: Bearer $SPIRAL_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "gpu",
    "capabilities": {
      "architecture": "x86_64",
      "operatingSystem": "linux",
      "resources": {"cpuCount": 8, "memoryBytes": "34359738368"},
      "installedTaskSpecs": [{"name": "spiral.table.maintenance", "version": 2}]
    },
    "maxWorkers": 10
  }'

The response contains the workerPool with its workerPoolId and status. Without maxWorkers, the limit is 1000.

Start each Worker with the pool's name or ID:

spiral-host worker start --project demo --pool gpu

The pool must be in the same Project. Each Worker must support every Task specification the pool declares, and its resources must cover the pool's.

Size the PoolCopy Anchor Link

Read the pool to see its live statistics:

curl https://api.spiraldb.dev/api/tasks/projects/demo/worker-pools/$WORKER_POOL_ID \
  -H "Authorization: Bearer $SPIRAL_TOKEN"

stats contains claimedReadyTaskCount, runningTaskCount, activeWorkerCount, busyWorkerCount, and the time of the oldest ready Task. To decide how many Workers to run, do what Spiral's autoscaler does for managed pools:

wanted Workers = claimedReadyTaskCount + busyWorkerCount

claimedReadyTaskCount counts the waiting Tasks this pool has reserved; while the pool holds them, no other pool counts them. busyWorkerCount counts Workers running an Attempt, including ones finishing before they exit. Start the difference between the wanted number and activeWorkerCount. An idle Worker takes the next Task itself, so you never need a new Worker while one is idle.

Spiral's autoscaler reads the same number from GET .../worker-pools/{worker_pool_id}/demand, but that endpoint does not accept workload tokens, so use stats.

Pause or retire the PoolCopy Anchor Link

Fix connection errors behind NATCopy Anchor Link

If Workers behind one NAT gateway start to see storage connection errors or timeouts, they may be using up the gateway's ports. Set these variables in each Worker's environment to limit its storage traffic:

The limits apply to each client in each Worker. There is no shared limit across Workers or machines, so lower them further as you add Workers.

TroubleshootingCopy Anchor Link

Where next: run a GPU Worker, or inspect the Tasks and Workers of the pool.