ScanDAL

The ScanDAL service runs Table scans, writes, blob transfer and SPQL evaluation for your application, next to the data, so your application needs no native library to read or write a Table. It is the hosted form of ScanDAL, the same operations that embedded ScanDAL runs inside your process, and it returns the same results. These HTTP operations are part of Connect; the embedded implementation is libspiral.

An exampleCopy Anchor Link

To read the front-camera rows of frames@main, the Rust SDK resolves frames@main to an exact version, gets storage access for it from the Tables service, and sends both with the scan to https://scandal.spiraldb.dev. ScanDAL plans the scan into splits, one for each file that the filter does not rule out, and streams each split back as Arrow rows. Select the service with scandal_url on the client builder; the rest of your code does not change.

Every request carries a ScanDAL capability: your Spiral access token, narrowed to ScanDAL and your stack's API, so that ScanDAL can read the blobs your rows address. Listing the functions the service can run needs nothing else:

export SCANDAL_TOKEN="$(curl -s https://auth.spiraldb.com/oauth/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:token-exchange \
  -d subject_token="$SPIRAL_TOKEN" \
  -d subject_token_type=urn:ietf:params:oauth:token-type:access_token \
  -d resource=https://scandal.spiraldb.dev \
  -d resource=https://api.spiraldb.dev | jq -r .access_token)"

curl https://scandal.spiraldb.dev/functions \
  -H "Authorization: Bearer $SCANDAL_TOKEN"

Why each request stands aloneCopy Anchor Link

A request names an exact Table version and carries the storage access it needs, so ScanDAL looks up no names and needs no state from an earlier request. Your application, or the SDK on its behalf, makes those control-plane decisions first, as it does for embedded ScanDAL. Because no request depends on an earlier one, any instance can serve it: a client reads splits in parallel, and resumes an interrupted split elsewhere by sending it again from the last row it received.

While your storage access remains valid, ScanDAL can reuse Table and file metadata for requests with the same access. This reuse is optional: each request can also run on an instance that has read none of that metadata.

The capability and the storage access answer different questions. The capability tells ScanDAL who the work is for, so the service refuses a request without a valid one before it does any work. The storage access decides which data the request can reach: SPFS checks it on every object request, and ScanDAL holds no storage credential of its own. A capability expires with the token it narrows and names only ScanDAL and your stack's API, so ScanDAL cannot use it to reach another service as you.

Rows can also address blobs in other resources by spiral:// address, such as a file under a Location. The request carries no storage access for those. When a read meets one, ScanDAL presents your capability to your stack's storage-binding route for that resource, and the stack checks that you may read that resource, whichever Table addressed it. A capability requested for ScanDAL alone still reads the Table, but fails on the first spiral:// blob.

An Application acting for a user gets a capability the same way. Its sign-in requests ScanDAL as a resource beside the stack's API, which every Application may, and it narrows the user's token to both. The capability keeps the token's scopes and names the Application as its client. ScanDAL does not check the scopes: the Tables service applies them when it issues storage access, so the Application reaches only the data the user let it read or write.

Source field identitiesCopy Anchor Link

A scan plan exposes sourceFields, a map from top-level stored names to stable field identities. The map follows the exact Table version and remains independent of the output schema. An absent map means that the host does not expose stable identities.

Reading legacy keyed tablesCopy Anchor Link

Studio uses ScanDAL to browse legacy keyed tables. The client resolves a snapshot and obtains storage access from the Tables API, then sends both to POST /legacy-tables/read. The response is an Arrow IPC stream with up to 10,000 rows plus one row indicating that more results exist. Use a zero-row limit to read the schema. The scan engine validates filter operations against column types. Ordered comparisons can filter text keys; for example, a greater-than filter can select keys after the last returned key. Filters and an offset select the window; refreshing in Studio resolves a new snapshot.

Use filter to combine field references, typed literals, comparisons, null checks and synchronous SPQL calls with AND, OR and NOT. ScanDAL returns InvalidArgument for asynchronous calls and other expression forms. ScanDAL applies this expression together with filters before the row limit and offset. Set columns to read only the top-level fields your application needs, in the requested order. Predicates can reference fields that you omit from columns. Omit columns to read every field.

At most two legacy reads execute at once and two more wait for a slot. A waiting read has one second to acquire a slot, then a separate 60-second execution budget. Waiting does not occupy the server capacity used by other operations. If the queue is full or no slot becomes available within one second, the request returns Unavailable and can be retried with backoff. Invalid storage bindings are rejected before waiting. Blocking session setup and image preparation retain their slots until they finish, even if the request times out. ScanDAL processes image presentations one row at a time. Each decoded image must fit within 64 MiB of pixel data, including when requesting a thumbnail. The rendered image buffers must fit in the remaining response budget. The serialized Arrow IPC response is limited to 64 MiB, reduced when its host sets a smaller memory budget. Large source images within the pixel budget can still produce small thumbnails. ScanDAL compacts string and binary views so unselected values in shared buffers do not count toward the response limit. Exceeding either budget fails the read; narrow the filter before retrying. These limits do not cap all temporary memory used to decode stored data or prepare previews. Loading media from URI references is not supported by this operation.

Legacy previews require support for read snapshots in the Tables API and legacy reads in ScanDAL. A client update alone does not enable this operation.

Hosted ScanDAL reads cloud storage. Reading a local file requires ScanDAL running on your machine, through desktop Studio or a paired spiral connect process. This operation is read-only and preserves the legacy table's format.

Similarity searchCopy Anchor Link

Your application can search an exact Vector Index through POST /vector-indexes/search. The request carries the index artifact and storage access for both Resources. The artifact origin supplies the exact source Table version. The manifest must match the requested index, version, source, and embedding field identities. An artifact without these identities requires a refresh.

query supplies the vector, and limit selects up to 1,000 results. probes selects the number of partitions, and rerank caps candidates at 100,000. ScanDAL applies the source filter and excludeRowIds before the final top-k selection. A filter can produce fewer results than limit because the candidate budget stays fixed.

The Arrow response contains scandal.row_id, scandal.distance, and the one-based scandal.rank. Requested source columns follow these fields. Omission of columns returns only these three fields. Final distances use float64 source coordinates, and candidate selection uses float32 coordinates. Equal distances sort by stable row ID. Dot distance is the negative inner product, so every metric sorts smaller distances first.

The schema metadata key scandal.vector_search contains the exact identities and effective candidate limits as JSON. Its exhaustive flag means that every indexed row fits within both limits. A shorter result proves exhaustion of the filtered population only when this flag is true. One request returns one bounded result set, without a continuation token.

Search shares the service admission and memory budgets with scans. Each search retains at most 64 MiB of serialized index payloads. Exact refinement prefetches at most two source splits under the request memory budget. Final metadata reads remain sequential.

Each host also reserves up to 128 MiB for reusable data from exact index versions. This data includes parsed manifests and prepared code planes. Every cache hit requires an object read with the current storage credentials. A strong object validator permits a one-byte read if its value still matches. Other hits require a complete read and checksum verification. Active requests retain their own memory costs after cache eviction.

The operation reads index payloads only after metadata and source validation. It reads requested source columns only for final results. A budget failure returns BudgetExceeded. An HTTP failure after the response starts truncates the Arrow stream, without its end marker. Stream cancellation releases the request reservations.

Top-k and aggregatesCopy Anchor Link

A facet list or a dashboard tile needs a few numbers or the ten newest frames, not every row of frames. Add an emit to the scan, and ScanDAL returns those in place of the rows, so the rows never leave the service. This emit counts the frames of each camera and finds each camera's first frame:

{
  "type": "aggregate",
  "aggregate": {
    "groups": ["camera"],
    "aggregates": [
      { "name": "frames", "function": { "type": "count", "count": {} } },
      { "name": "first_seen", "function": { "type": "min", "min": { "column": "captured_at" } } }
    ],
    "maxGroups": 20
  }
}

Each split returns one row for each camera it read, with frames, first_seen, scandal.rows (the rows the camera covers) and scandal.other. These are partial results, and merging every split's rows gives the answer: counts and sums add, minimums and maximums compare, and histograms add bin by bin. A sum of no values is zero, and an integer sum that overflows is null. When a split has no filter and no deleted rows, counts, and minimums and maximums of numbers, come from statistics the Table keeps for each file, so the split reads no rows. The Rust and TypeScript SDKs read the splits in parallel and merge them for you when you set emit on a scan. A split's results are complete only when its stream ends, so a client reads a failed split again from its start.

Aggregates keep at most maxGroups groups, 1,000 unless you set it. Rows of further groups fold into one row whose scandal.other is true, so the result stays small however many distinct values a column holds. While the Table has no more distinct groups than maxGroups, the result is exact. A merged result lists groups by their rows, most first.

A top-k emit returns the k rows that sort first by its keys, with every output column. For the ten newest frames, the emit is { "type": "topK", "topK": { "keys": [{ "column": "captured_at", "descending": true }], "k": 10 } }. Each split reads only the key columns to choose its winners, then reads the other columns of those ten rows, so a wide image column is read for ten frames, not for all of them. Once a split holds ten frames, it skips frames, and whole stretches of the file whose statistics show them, that are older than its tenth newest, so a top-k over a sorted column reads little more than its winners. k is at most 10,000. To sort a whole Table, use your own query engine.

The row ID column can supply a stable tie breaker for a top-k. Aggregates do not return per-row identities. The Python client uses top_k on a scan.

page = frames.scan(
    ["frame_id", "camera", "captured_at"],
    row_id_column=spiral.ROW_ID_COLUMN,
    top_k=spiral.TopK(
        50,
        [
            spiral.SortKey("captured_at", descending=True),
            spiral.SortKey(spiral.ROW_ID_COLUMN),
        ],
    ),
).to_table()

Reading rows by positionCopy Anchor Link

A sampler that draws rows at random, such as a data loader's shuffled epoch, reads them with POST /scans/take. The request carries the scan, the storage access, and takes: each a split of the scan's plan and positions within it. Position p is the p-th row of the split, in stored order, that the Table version keeps after deleted rows. These takes read three rows of one split and one of another:

{
  "takes": [
    { "split": { "fragment": 3, "rows": { "start": 0, "end": 4096 } }, "positions": [812, 17, 812] },
    { "split": { "fragment": 7, "rows": { "start": 0, "end": 4096 } }, "positions": [2048] }
  ]
}

ScanDAL returns the rows in request order, the row at position 812 twice, as an Arrow stream framed like a split read: scandal.next_row counts the positions delivered so far, so a client resumes by sending the positions after them. It reads each split's rows in one pass, so positions that lie close together in storage share their reads. A request holds at most 65,536 positions from at most 256 splits, and its scan has no filter, limit, or emit. A position past the rows of its split fails the request before any rows are sent. The Rust SDK counts each split's rows and sends the takes for you.

How much one request may useCopy Anchor Link

Each request may hold 512 MiB of memory: the batch it is sending and the state of its emit, such as a top-k's winners or an aggregate's groups. A request that needs more fails alone with BudgetExceeded, and the other requests on the same instance carry on. Narrow it with a smaller k or maxGroups, or fewer or narrower columns.

An instance runs as many requests at once as their budgets fit in its memory. When it is full, it refuses a request with Unavailable, HTTP status 503, before doing any work, and its Retry-After header says how many seconds to wait. The SDKs wait and send the request again.

Uploading large blobsCopy Anchor Link

A video or a point cloud can be larger than any one request should carry, so ScanDAL uploads it in parts that any instance accepts. Each part is a request of its own: a failed part is sent again by itself, and parts of one upload can go to different instances at once. The upload belongs to one write of a Table, and the blob it stores becomes part of the Table only when a row of that write references it and the write commits.

1. POST /blobs/uploads/start takes the Table, the write, and the write's storage access, and returns an upload: the Table, the write, the blob's object ID, the storage provider's upload ID, and the part size, 64 MiB unless the request sets a power of two from 8 MiB to 1 GiB.
2. POST /blobs/uploads/part stores one part. The scandal-request header carries the upload and the part's number, counted from 1, and the body carries its bytes. Every part but the last holds exactly the part size. The response holds the part's storage tag and the BLAKE3 hash of its share of the payload.
3. POST /blobs/uploads/complete takes every part's response, in order, and returns the blob. Its digest is the BLAKE3 digest of the whole payload, which ScanDAL merges from the parts' hashes without reading a byte again.
4. Append a row whose blob column carries the blob, in the same write.

The upload keeps no state in ScanDAL: the storage provider holds its parts, and your client holds the upload and the part responses. POST /blobs/uploads/abort discards an unfinished upload's parts; aborting removes no stored object. Each part counts against the request's memory budget, so a part is at most 512 MiB. The Rust and TypeScript SDKs run these steps for you.

Packing small blobsCopy Anchor Link

Many small payloads, such as camera frames, are cheaper to read from a few large objects than from one object each. POST /blobs/pack takes the Table, an optional write, and an Arrow stream whose spql.Blob columns hold inline bytes. It stores those payloads as a write stores the inline bytes of the rows it appends: in packs of up to 64 MiB under the Table's blobs/ directory, or the write's, with a payload of 4 MiB or more as an object of its own and equal payloads sharing one range. It returns the same rows with each payload replaced by its address, offset, size and BLAKE3 digest, once every pack is stored. Append rows that carry those blobs, in the same write. Each Arrow batch of the request holds at most 64 MiB; upload a larger payload in parts instead. The Rust SDK's blob pack sends this request for you.

Calling ScanDAL from a browserCopy Anchor Link

Use the TypeScript SDK's connect() to authenticate with a short-lived capability and establish a secure, HttpOnly, partitioned cookie. Subsequent Table operations name resources and exact versions; the service obtains scoped storage access as that caller. Hosted writes obtain separate create-only access for their attempt and publication.

The Application registers each page's exact origin through allowedCorsOrigins. The authorization server embeds that configuration in the signed token. ScanDAL verifies the signature and checks the request's Origin against that list before work. Origin changes take effect when the token is renewed, bounded by its lifetime. OPTIONS preflight permits origins without authentication and sets no cookie. Actual responses expose credentials only to configured origins, with Vary: Origin.

Cookies are host-only and partitioned by the top-level site. They expire with the capability, require an allowed Origin, and require x-spiral-request: 1 on state-changing requests. An explicit Authorization header takes precedence. The service keeps no session registry or affinity; every replica verifies the same signed capability using the authorization server's published keys.

GET /files and HEAD /files serve resource-owned blobs, including packed slices, to native images and video. A single HTTP Range is relative to the logical slice; ETag and version pins constrain the upstream object. Set the element's crossOrigin to use-credentials before setting its URL. Responses are private and never cached. The HTTP reference covers the cookie bootstrap, renewal, logout and byte-range protocol.

Storage bindings are optional on the existing operations. When omitted, the service obtains access as the authenticated caller, scoped to the read, write attempt or publication. Clients with explicit bindings and paired local Connect use the same routes.

Where it runsCopy Anchor Link

Spiral runs the service in us-east4 at https://scandal.spiraldb.dev, next to Spiral-managed storage in that region. Rows and blob bytes pass through the service, but not through Spiral's control plane.

AvailabilityCopy Anchor Link

The service runs in us-east4 only; the other platform regions are planned. It reads and writes through SPFS or scoped cloud provider credentials. The Rust SDK uses the service only when you pass its endpoint.