Video encoding parameters

When video storage is enabled, LeRobot stores each camera stream as an MP4 file instead of saving one image file per timestep. Video encoding compresses across time, which usually cuts dataset size and I/O compared to a pile of PNG, while keeping MP4 — a format every player and loader understands.

Encoding frames into an MP4 is a full FFmpeg pipeline: choice of encoder, pixel format, GOP/keyframes, quality vs. speed, and optional extra encoder flags. Most of these knobs are user-tunable through rgb_encoder, a nested RGBEncoderConfig (lerobot.configs.video.RGBEncoderConfig) passed through PyAV.

You can set these parameters from the CLI with --dataset.rgb_encoder.<field> (e.g. with lerobot-record or lerobot-rollout). The same block applies to every camera video stream in that run.

Video storage must be on for rgb_encoder to have any effect — use_videos=True in Python APIs, or --dataset.video=true on the CLI (the recording default). With video off, inputs stay as images and rgb_encoder is ignored.

For details on when frames are written vs. encoded (streaming vs. post-episode), queues, and other top-level --dataset.* switches, see Streaming Video Encoding. For an encoding-parameter comparison and experiments, see the video-benchmark Space.

Example

lerobot-record \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem58760431541 \
    --robot.cameras="{laptop: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --robot.id=black \
    --teleop.type=so100_leader \
    --teleop.port=/dev/tty.usbmodem58760431551 \
    --teleop.id=blue \
    --dataset.repo_id=<my_username>/<my_dataset_name> \
    --dataset.num_episodes=2 \
    --dataset.single_task="Grab the cube" \
    --dataset.streaming_encoding=true \
    --dataset.encoder_threads=2 \
    --dataset.rgb_encoder.vcodec=h264 \
    --dataset.rgb_encoder.preset=fast \
    --dataset.rgb_encoder.extra_options={"tune": "film", "profile:v": "high", "bf": 2} \
    --display_data=true

Tuning parameters

The defaults are tuned to balance compression ratio, visual quality, and decoding/seek speed for typical robotics datasets. Changing them can affect both recording (CPU load, frame drops) and training (decoding throughput, image quality).

Only override these parameters if you have a specific reason to, and measure the impact on your pipeline before relying on the new settings.

All flags below are prefixed with --dataset.rgb_encoder. on the CLI.

Depth streams

Depth maps (Intel RealSense, Reachy 2) are stored as their own video streams alongside the RGB streams. Raw depth (uint16 millimetres or float32 metres) can’t survive an 8-bit codec, so LeRobot quantizes each map to a 12-bit code ([0, 4095]) — logarithmically by default, to match the 1/depth error profile of depth sensors — then packs it into a high-bit-depth pixel format (gray12le) and encodes it with a 12-bit codec.

Configure the depth pipeline through a parallel depth_encoder block (DepthEncoderConfig). It shares every RGBEncoderConfig field (vcodec, pix_fmt, crf, …) and adds four quantizer knobs, set via --dataset.depth_encoder.<field>:

lerobot-record \
    ... \
    --dataset.depth_encoder.vcodec=hevc \
    --dataset.depth_encoder.depth_min=0.05 \
    --dataset.depth_encoder.depth_max=5.0 \
    --dataset.depth_encoder.use_log=true

depth_min, depth_max, and shift are always interpreted in metres, regardless of the input depth’s unit. Inputs are auto-detected: integer arrays (e.g. uint16 millimetres straight from a RealSense) are treated as millimetres, floating arrays as metres. Pick depth_min / depth_max to bracket the actual working range of your sensor — quanta outside that range saturate, which can crush detail at the boundaries.

Depth features are flagged with "is_depth_map": true in meta/info.json, and their quantizer settings (video.depth_min, video.depth_max, video.shift, video.use_log) are persisted — which is what lets depth be dequantized back to physical units on load.

Output unit at load time

depth_encoder is a record-time concern. The unit that depth maps are dequantized to on load (e.g. during training) is set separately by the read-time flag --dataset.depth_output_unit:

lerobot-train \
    --dataset.repo_id=<my_username>/<my_dataset_name> \
    --dataset.depth_output_unit=m \
    --policy.type=act

This is purely a decode-time presentation choice — it does not alter the stored video or its metadata, so the same dataset can be read as mm or m without re-encoding. It has no effect on datasets without depth cameras.

Persistence in dataset metadata

After the first episode of a video stream is encoded, the encoder configuration is persisted into the dataset metadata (meta/info.json) under each video feature, alongside the values probed from the file itself. For a video feature observation.images.<camera>, the layout in info.json is:

{
  "features": {
    "observation.images.laptop": {
      "dtype": "video",
      "shape": [480, 640, 3],
      "info": {
        "video.height": 480,
        "video.width": 640,
        "video.codec": "h264",
        "video.pix_fmt": "yuv420p",
        "video.fps": 30,
        "video.channels": 3,
        "is_depth_map": false,
        "video.g": 2,
        "video.crf": 30,
        "video.preset": "fast",
        "video.fast_decode": 0,
        "video.video_backend": "pyav",
        "video.extra_options": { "tune": "film", "profile:v": "high", "bf": 2 }
      }
    }
  }
}

Two sources contribute to the info block:

This block is populated once, from the first episode. It assumes every episode in the dataset was encoded with the same rgb_encoder. Changing encoder settings partway through a recording is not supported — the info.json will only reflect the parameters used for the first episode.

Merging datasets

When aggregating datasets with merge_datasets, video files are concatenated as-is (no re-encoding), and encoder fields in info.json are merged per-key: