LeRobot Import Mapping

A LeRobot v3 dataset imports as one element per episode. Each episode holds its frames, with every feature typed as the dataset's meta/info.json declares it, and one video clip per camera that points into the dataset's own MP4 files. Your application reads joint states as tensors and seeks camera video by episode time, instead of joining the dataset's Parquet and MP4 files itself. The mapping is fixed by info.json, so every dataset with the same features imports with the same episode type.

A dataset with an observation.state vector, an action vector and one observation.images.front camera imports episodes such as:

episode_index  1
tasks          [place the cube, wave]
length         6
frames         [{observation.state: [12.0, 13.0], action: [0.6, -1.0], timestamp: 0.0,
                 frame_index: 0, episode_index: 1, index: 6, task_index: 1}, …]
videos         {observation.images.front: {clip: {media: videos/…/file-000.mp4,
                 start: 1s, end: 2s}, time_origin: -1s}}

EpisodesCopy Anchor Link

LeRobot v3 packs many episodes into each data file and each video file, and records where every episode lies in meta/episodes/*/*.parquet. An episodes metadata file imports as a list of episodes, one for each of its rows, in the file's order.

Feature names are kept as LeRobot writes them, including their dots, so observation.state is the field observation.state of each frame. The dataset's other files are found from the paths in info.json, relative to the folder that holds meta/info.json.

Feature mappingsCopy Anchor Link

A feature's dtype and shape in info.json select its type. A frame field is null where the data file holds a null.

The default features timestamp, frame_index, episode_index, index and task_index are frame fields like any other. timestamp is the frame's time in seconds from the start of its episode, and it is the episode's timeline.

TensorsCopy Anchor Link

A numeric feature of any shape other than [1] imports as a fixed-shape tensor, so a [2, 3] feature holds six values per frame in row-major order: the three values of the first row, then the three of the second. LeRobot writers store a tensor as nested lists, one level for each dimension, either fixed-size or holding exactly the declared number of values, and both read the same way. A frame whose lists hold any other number of values fails the read.

The tensor's dimension names come from the feature's names when it names each dimension once:

• a list with one name per dimension, such as ["finger", "axis"] for a [2, 3] feature
• an object with one key per dimension, such as {"finger": ["left", "right"], "axis": ["x", "y", "z"]}, whose keys name the dimensions in order and whose lists name each dimension's entries

Other names are not kept. The common list of one name per element of a vector, such as ["x", "y"] for a [2] feature, names the elements rather than the dimension, so a vector imported with it has no dimension name.

ImagesCopy Anchor Link

An image feature holds each frame's encoded image, as LeRobot writes it in the data files: a struct of the image's bytes and a path. It imports as a spql.EncodedImage whose source is a spql.Blob holding the bytes inline, so the image stays encoded until a reader decodes it.

The import does not read image headers. The image's width and height are the entries of the feature's shape that its names call width and height, such as 160 and 90 for a [90, 160, 3] feature with names ["height", "width", "channels"], and they are null when the feature's names do not name them. format and mime_type are null; decoding finds the format from the bytes. A frame whose struct holds no bytes, such as an image stored only by path, imports as a null image.

Video clipsCopy Anchor Link

A video feature's clip is a spql.ClipRef over the MP4 file that holds the episode:

• media is a spql.EncodedMedia whose source is the file's URI with its size, and whose container is mp4.
• start and end are the episode's from_timestamp and to_timestamp, the half-open range of the episode in the file's own clock.
• stream_indices is null, which selects every stream of the file.

time_origin is the value of the episode's timestamp at which the file's stream time is zero: the negative of from_timestamp, as a spql.Duration in nanoseconds. Add it to a stream time to place a frame on the episode's timeline, and subtract it from a timestamp to find the frame in the file. Each video feature has its own origin, because each feature's episodes are packed into its own files.

When the feature's video.codec is h264, hevc or av1, the media holds a spql.MediaIndex of the file, read from its moov box, so a reader can seek to any frame of the episode without opening the file first. LeRobot v3 datasets usually encode AV1. Other codecs import with a null index. The index covers the whole file, so every episode in the file carries the same index.

ErrorsCopy Anchor Link

Reading an episodes file fails when info.json is not codebase_version v3.0, when a bool or string feature has a shape other than [1], when a data file's index column is not consecutive or does not hold an episode's frames, or when a column does not convert to its declared type. An image column that is not a struct with binary bytes, such as one that holds decoded pixels, also fails. A metadata file whose episodes span more than 64 MiB of data files cannot be read as one value yet.

AvailabilityCopy Anchor Link

The import that writes episodes into a Table is planned, and so are reads of large datasets in parts. Until they ship, no public client exposes this mapping.