PyTorch

The PyTorch integration turns a scan of an exact Table version into a torch.utils.data.IterableDataset. A training job reads Table rows straight from storage, each rank and DataLoader worker reads only its own rows, and a restarted job resumes from a saved position instead of reading the epoch again. It is the spiral.torch module of the Python SDK, installed with the torch extra, spiral-sdk[torch].

How rows are shared outCopy Anchor Link

A scan of the frames Table is planned as splits: ranges of stored rows in the exact version. Each DataLoader worker on each rank reads an ordered list of work items, where an item is a split or a range of its rows. By default, the dataset stripes whole splits: with two ranks, rank 0 reads splits 0, 2, 4 and so on, and each rank's workers take turns through that rank's splits.

Striping depends only on the splits, the rank, the world size and the worker, so every process computes the same lists without coordinating, and no rank reads another rank's rows. A rank reads the same splits whatever its worker count. Splits of different sizes give workers different numbers of rows, and a filter can remove more rows from one worker's splits than another's.

The lists come from a work policy, a function of the scan and the worker's place among all workers. To read the rows in another order, such as a shuffle, pass your own policy; reading, resuming and counting work the same way for any list.

The dataset opens the scan only inside the process that reads it, so it is safe to pass to DataLoader workers. Each worker frames its rows into batches of a fixed size. With last="keep_partial", the rows after a worker's last full batch arrive as one shorter batch; with last="require_full", they raise an error instead. Use the dataset with DataLoader(batch_size=None), because the batches are already framed. Give a DataLoader with workers multiprocessing_context="forkserver" or "spawn". A worker forked from a process that has already used Spiral cannot continue, so it raises an error instead of hanging; Linux starts workers by fork unless told otherwise.

Resume a runCopy Anchor Link

state_dict and load_state_dict follow torchdata's StatefulDataLoader. The saved state is a cursor into the worker's share: the split range it was reading, the stored row where its undelivered rows begin, and the rows already delivered from there. A restored dataset opens that range at that row and skips only within it, so resuming costs the same whether the job stopped early or late in the epoch. The state also records the version, the worker's place, the batch size and a fingerprint of its work list, and restoring it with different settings fails.

len(dataset) is the exact number of rows the rank reads. The dataset counts them once, reading only the columns the filter needs.

TensorsCopy Anchor Link

Numeric columns and fixed-size lists of numbers become tensors that share memory with the scan's output, through DLPack. Images projected with spql.image.to_tensor arrive as one tensor for each batch, as HWC uint8 or CHW float32 in the layout the projection requests, when the batch's images share a size, which spql.image.resize ensures. Video clips projected with spql.video.to_tensor over the frames spql.video.read selects arrive the same way, one [T, H, W, C] or [T, C, H, W] tensor per row, stacked into [B, T, …] when the batch's clips have the same length. Strings and ragged lists stay Python lists.

Tensors are in host memory. Move them with pin_memory=True on the DataLoader and non_blocking=True copies. Video decoding runs on the CPU through FFmpeg.

AvailabilityCopy Anchor Link

The integration ships with the Python SDK; install spiral-sdk[torch] to include PyTorch. It supplies no shuffle or other ordering policy of its own yet. Sample orders, mixtures of Tables, time windows, and decoding on the GPU are planned.