Video
Inspect encoded media and MP4 structure, select video frames, and decode images.
Scalar Functions
| Function | Summary |
|---|---|
spql.media.clip | Construct a half-open encoded-media selection without reading bytes. |
spql.media.from_blob | Construct encoded media from a blob without reading bytes. |
spql.media.media_index | Construct a spql.MediaIndex value. |
spql.media.stream | Select a stream by its structural container index without reading bytes. |
spql.video.decode | Decode one FrameRef to canonical display-oriented RGB8 Image pixels. |
spql.video.decode_yuv420 | Decode one FrameRef to separate Y, U, and V 4:2:0 image planes. |
spql.video.frame_at | Resolve a presentation time to one FrameRef. |
spql.video.frame_indices | Construct a spql.FrameIndices value. |
spql.video.frame_neighborhood | Construct a spql.FrameNeighborhood value. |
spql.video.frame_slice | Construct a spql.FrameSlice value. |
spql.video.frame_times | Construct a spql.FrameTimes value. |
spql.video.from_packets | Describe a video stream from explicitly selected packet payloads. |
spql.video.image_sequence | Construct a fixed-rate encoded image sequence. |
spql.video.index | Index an MP4 blob's video tracks for frame-exact reads. |
spql.video.read | Resolve a frame selection to the stream's presentation-order FrameRefs. |
spql.video.to_tensor | Decode a list of FrameRefs to one clip tensor. |
spql.media.clip
Construct a half-open encoded-media selection without reading bytes.
- Kind:
scalar function - Volatility:
immutable
spql.media.clip(media, stream_indices?, start?, end?) -> spql.ClipRef?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
media | yes | spql.EncodedMedia? | EncodedMedia containing the tracks to select. |
stream_indices | no | list(i32)? | Container track indexes to include; omitted selects all tracks. |
start | no | spql.Duration? | Inclusive start time of the selection; omitted selects from the beginning. |
end | no | spql.Duration? | Exclusive end time of the selection; omitted selects through the end. |
spql.media.from_blob
Construct encoded media from a blob without reading bytes.
- Kind:
scalar function - Volatility:
immutable
spql.media.from_blob(source, container?, index?) -> spql.EncodedMedia?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
source | yes | spql.Blob? | Blob containing the encoded media payload. |
container | no | utf8? | The container, such as mp4. |
index | no | spql.MediaIndex? | The media's index, such as spql.video.index computes. |
spql.media.media_index
Construct a spql.MediaIndex value.
- Kind:
scalar function - Volatility:
immutable
spql.media.media_index(primary_track_id, tracks) -> spql.MediaIndex?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
primary_track_id | yes | u32? | Identifier of the primary media track. |
tracks | yes | list(struct { track_id, kind, codec, language, width, height, timescale, movie_timescale, duration_ticks, sample_entry, dts_runs, edit_list, chunks, gops, sizes, ctts, flags })? | Track metadata, sample tables and GOP indexes for the media. |
spql.media.stream
Select a stream by its structural container index without reading bytes.
- Kind:
scalar function - Volatility:
immutable
spql.media.stream(media, index) -> spql.StreamRef?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
media | yes | spql.EncodedMedia? | EncodedMedia containing the tracks to select. |
index | yes | i32? | Zero-based container track index. |
spql.video.decode
Decode one FrameRef to canonical display-oriented RGB8 Image pixels.
- Kind:
scalar function - Volatility:
immutable
spql.video.decode(frame, 'decoder'?) -> spql.Image?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
frame | yes | spql.FrameRef? | FrameRef identifying the encoded frame to decode. |
decoder | no | utf8 | Decoder implementation: auto, ffmpeg, videotoolbox, or nvdec. |
spql.video.decode_yuv420
Decode one FrameRef to separate Y, U, and V 4:2:0 image planes.
- Kind:
scalar function - Volatility:
immutable
spql.video.decode_yuv420(frame, 'decoder'?) -> spql.Yuv420Image?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
frame | yes | spql.FrameRef? | FrameRef identifying the encoded frame to decode. |
decoder | no | utf8 | Decoder implementation: auto, ffmpeg, videotoolbox, or nvdec. |
spql.video.frame_at
Resolve a presentation time to one FrameRef.
- Kind:
scalar function - Volatility:
immutable
spql.video.frame_at(source, time, mode?) -> spql.FrameRef?Reads no pixels. A stored MediaIndex answers without reads; otherwise the MP4 header boxes are indexed. EncodedMedia selects its primary video track. Nearest breaks ties toward the earlier frame. Nearest, floor and ceil clamp to the stream's endpoints; exact requires a matching presentation time. A null source or time yields null, and a null mode means nearest. Packet sources inspect complete H.264 Annex-B access units with an initial IDR and SPS/PPS configuration. They require presentation times on every picture packet and reuse their header index within the execution.
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
source | yes | list(struct { decode_index, packet })?, spql.EncodedMedia?, spql.StreamRef? | An EncodedMedia value, a StreamRef naming one track, or an H.264 Annex-B packet source. |
time | yes | spql.Duration? | Presentation time from the stream's start. |
mode | no | utf8? | nearest (default), exact, floor/previous, or ceil/next. |
spql.video.frame_indices
Construct a spql.FrameIndices value.
- Kind:
scalar function - Volatility:
immutable
spql.video.frame_indices(indices) -> spql.FrameIndices?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
indices | yes | list(i64)? | Display-order frame positions. Negative positions count from the end. |
spql.video.frame_neighborhood
Construct a spql.FrameNeighborhood value.
- Kind:
scalar function - Volatility:
immutable
spql.video.frame_neighborhood(time, count, start_offset?, step?, mode?, boundary?) -> spql.FrameNeighborhood?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
time | yes | spql.Duration? | Presentation time used to resolve the anchor frame. |
count | yes | u32? | Number of output frame references. |
start_offset | no | i64? | First frame offset from the anchor. A null value centers the neighborhood. |
step | no | i64? | Nonzero frame-position increment. A null value means one. |
mode | no | utf8? | Anchor resolution mode: nearest, exact, floor, or ceil. A null value means nearest. |
boundary | no | utf8? | Boundary mode: clamp, truncate, or error. A null value means clamp. |
spql.video.frame_slice
Construct a spql.FrameSlice value.
- Kind:
scalar function - Volatility:
immutable
spql.video.frame_slice(start?, stop?, step?) -> spql.FrameSlice?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
start | no | i64? | Inclusive start position. A null value selects from the first frame. |
stop | no | i64? | Exclusive stop position. A null value selects through the last frame. |
step | no | i64? | Nonzero position increment. A null value means one. |
spql.video.frame_times
Construct a spql.FrameTimes value.
- Kind:
scalar function - Volatility:
immutable
spql.video.frame_times(times, mode?) -> spql.FrameTimes?Parameters
| Name | Required | Type | Description |
|---|---|---|---|
times | yes | list(spql.Duration)? | Requested presentation times. |
mode | no | utf8? | Resolution mode: nearest, exact, floor, or ceil. A null value means nearest. |
spql.video.from_packets
Describe a video stream from explicitly selected packet payloads.
- Kind:
scalar function - Volatility:
immutable
spql.video.from_packets(packets, codec, framing) -> list(struct { decode_index: i64, packet: spql.EncodedPacket? })?A null packets, codec or framing value produces null; an empty list produces an empty source. Sort descriptors by decode_index, preserving optional presentation times. Null list elements, null payloads, null or negative decode indices, duplicates and unsupported codec/framing values are errors. Construction reads no payloads and decodes no pixels: Blob locators stay external and binary payloads stay inline. The caller supplies the topic selection and timestamp mapping.
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
packets | yes | list<struct { payload, decode_index, presentation_time?, .. }> | A list of complete access-unit descriptors: payload (binary or Blob), unique nonnegative decode_index (i64), and optional presentation_time (nanosecond duration). |
codec | yes | utf8? | Packet codec; h264 is supported. |
framing | yes | utf8? | Packet framing; annex_b is supported. |
Returns
A list of decode-ordered EncodedPacket descriptors.
spql.video.image_sequence
Construct a fixed-rate encoded image sequence.
- Kind:
scalar function - Volatility:
immutable
spql.video.image_sequence(images, frame_rate) -> spql.ImageSequence?Pass the result to spql.viz.video_view for playback. Use frame relations for unbounded or variable-rate video.
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
images | yes | list(spql.EncodedImage?)? | Encoded images in presentation order. |
frame_rate | yes | f64? | Positive playback rate in frames per second. |
spql.video.index
Index an MP4 blob's video tracks for frame-exact reads.
- Kind:
scalar function - Volatility:
immutable
spql.video.index(blob) -> spql.MediaIndex?Returns a spql.MediaIndex with each H.264, H.265 or AV1 track's sample entry, sample tables and GOPs, from which a reader derives byte offsets, presentation order and each frame's decode closure. A blob with a declared size is indexed from its ftyp and moov boxes alone, unless an H.264 track reorders its frames: then the whole blob is read for each sample's reference bit, so that closures leave out frames nothing references. Fragmented MP4 and blobs without a size are read whole. No sample is decoded. A null blob yields a null index. Store the index in the media value with spql.media.from_blob(blob, 'mp4', spql.video.index(blob)).
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
blob | yes | spql.Blob | An MP4 file, inline or addressed by URI. |
spql.video.read
Resolve a frame selection to the stream's presentation-order FrameRefs.
- Kind:
scalar function - Volatility:
immutable
spql.video.read(source, selection) -> list(spql.FrameRef?)?Reads no pixels. A media value's stored MediaIndex answers the selection without reads; otherwise the MP4 header boxes are indexed. EncodedMedia selects its primary video track. Each frame carries its presentation and decode positions, timestamps and keyframe flag. Out-of-range explicit indices and exact-time misses are errors; slices clamp to the stream. A null source or selection yields a null list. Packet sources inspect complete H.264 Annex-B access units, starting at an IDR with SPS/PPS configuration. Their header index is reused within one execution; time selections require explicit presentation times on every picture packet.
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
source | yes | list(struct { decode_index, packet })?, spql.EncodedMedia?, spql.StreamRef? | An EncodedMedia value, a StreamRef naming one track, or an H.264 Annex-B packet source. |
selection | yes | spql.FrameIndices?, spql.FrameSlice?, spql.FrameTimes?, spql.FrameNeighborhood? | A FrameIndices, FrameSlice or FrameTimes value; media and streams also accept FrameNeighborhood. |
spql.video.to_tensor
Decode a list of FrameRefs to one clip tensor.
- Kind:
scalar function - Volatility:
immutable
spql.video.to_tensor(frames, 1, 1, 'layout'?, 'normalization'?, 'decoder'?) -> spql.Tensor?Literal arguments can refine the result type, including its units, shape or element type.
Each frame is decoded, resized bilinearly to width by height and converted like spql.image.to_tensor, in list order. Without options the clip is a [T, H, W, C] u8 tensor; unit or imagenet normalization makes it a [T, C, H, W] float32 tensor. Frames of one media track decode in one pass, also across rows, and when the media carries its MediaIndex only the byte ranges of their decode closures are read. HWC clips decode on FFmpeg. For CHW clips, the decoder and the requested output device select FFmpeg on the CPU, VideoToolbox and Metal, or NVDEC and NPP. A null list yields a null tensor; a null FrameRef in a list is an error.
Parameters
| Name | Required | Type | Description |
|---|---|---|---|
frames | yes | list(spql.FrameRef?)? | FrameRefs to decode in list order into one clip tensor. |
width | yes | u32? | Output width in pixels. |
height | yes | u32? | Output height in pixels. |
layout | no | utf8 | Output layout: hwc for none or chw for unit and imagenet. |
normalization | no | utf8 | Pixel normalization: none, unit, or imagenet. |
decoder | no | utf8 | Decoder implementation: auto, ffmpeg, videotoolbox, or nvdec. |
Returns
One tensor per list, its first dimension the list's length.