Tasks

Some operations take minutes or hours: compacting a large Table, rebuilding a Vector Index, or computing an Enrichment over millions of frames. A Task lets your application hand that work to Spiral and move on. You submit it once, a Worker with enough CPU and memory picks it up, and Spiral retries it if the Worker dies and keeps a record of every run and its result. You can check on it later from the API or Studio, even if nothing stayed connected while it ran.

For example, after the nightly-curation job writes a batch of rows to the frames Table in the demo Project, it submits a Table maintenance Task to compact the Table, and exits:

{
  "definition": {
    "type": "tableMaintenance",
    "tableMaintenance": {
      "readTarget": {"type": "latestOnBranch", "latestOnBranch": {"branchId": "..."}},
      "tableResourceId": "..."
    }
  },
  "idempotencyKey": "maintain-frames-2026-10-13"
}

Spiral records the Task as ready. A Worker in the Project claims it and runs it, the Task moves to running, and it ends succeeded with the versions of the Table that it published. See Submit and manage Tasks for the full request.

What a Task isCopy Anchor Link

A Task starts in ready. It is running while an Attempt is active, and it ends succeeded, failed, or cancelled. When an Attempt fails in a way that can be retried, Spiral starts another one, up to the specification's attempt limit. The Task keeps one ID across all its Attempts, so you watch one thing no matter how many times it ran.

Task definitions and results are small JSON documents. A result lists the Resource versions that the Task published, nothing more. The Worker reads and writes Table data directly in storage, so your data never passes through the Task API.

Built-in TasksCopy Anchor Link

GET /api/tasks/projects/{project_id}/task-specs lists the specifications that a Project can run, with their default and allowed resource profiles.

A Task can name Resources as its target, sources, and dependencies. To submit one, you need the Project editor role and the permissions the specification requires on those Resources. Table maintenance, for example, needs read and write access to the Table.

Choosing a Worker sizeCopy Anchor Link

A resource profile sets the minimum Worker capacity for each Attempt:

Each specification has a default profile and a list of allowed ones. All built-in specifications allow all five and default to cpu_2. A submission without a profile uses the default. Spiral stores the resolved CPU and memory on the Task, so retries ask for the same capacity.

A profile is a minimum, not a limit: a larger Worker can run the Task. If no Worker that is large enough is available, the Task waits in ready. After six hours unclaimed, Spiral cancels it with the cancellationReason pending_timeout, so a missing Worker shows up as a visible outcome rather than a Task that waits forever. Inspect Tasks shows how much a Task actually used, so you can tell whether to pick a larger profile.

When two Tasks overlapCopy Anchor Link

Running two maintenance Tasks on the same Table at once wastes work, so some specifications limit how many Tasks can be active for the same Resource. The overlap policy decides what happens to a new submission while another Task for that Resource is active:

Table maintenance uses buffer_one for each Project, branch, and Table, so maintenance on other branches and Tables still runs concurrently. A skipped submission does not appear in Task lists, and a retry with the same idempotency key returns the same outcome. This is why you can submit maintenance after every write without piling up Tasks.

What happens when things failCopy Anchor Link

Spiral keeps a Task's state correct even when a Worker stops at the wrong moment:

• If a Worker stops responding, its Attempt times out and fails in a way that can be retried, so another Worker picks the Task up.
• If a Worker publishes its change but stops before it reports success, Spiral finds the published result and marks the Task succeeded without running it again.
• If you cancel a Task, Spiral first checks for a published result, which wins over the cancellation. Otherwise, Spiral blocks any late publication and marks the Task cancelled.

Each Attempt of a built-in Task can run for at most 48 hours.

How Tasks find WorkersCopy Anchor Link

You do not pick a Worker or a Worker Pool when you submit a Task. The Task declares what it needs, and a Worker in the same Project that can run that specification version with enough resources claims it. See How a Task finds a Worker.

AvailabilityCopy Anchor Link

Tasks are in development. Client support lists the clients that can submit and manage them.

Where it runsCopy Anchor Link

Spiral hosts the Task API at https://api.spiraldb.dev/api/tasks/projects/{project_id}/tasks. It records Tasks, Attempts, and Automations, and assigns Tasks to Workers. The Workers run on Spiral-managed compute or on your own, next to your storage. See Architecture.

GuidesCopy Anchor Link

• Submit and manage Tasks
• Automate Tasks on a schedule or when a Resource changes
• Inspect Tasks and Workers in Studio