LIBERO-plus

LIBERO-plus is a robustness benchmark for Vision-Language-Action (VLA) models built on top of LIBERO. It systematically stress-tests policies by applying seven independent perturbation dimensions to the original LIBERO task set, exposing failure modes that standard benchmarks miss.

• Paper: In-depth Robustness Analysis of Vision-Language-Action Models
• GitHub: sylvestf/LIBERO-plus
• Dataset: lerobot/libero_plus

Perturbation dimensions

LIBERO-plus creates ~10 000 task variants by perturbing each original LIBERO task along these axes:

Available task suites

LIBERO-plus covers the same five suites as LIBERO:

Installing LIBERO-plus **replaces** vanilla LIBERO — it uninstalls `hf-libero` so that `import libero` resolves to the LIBERO-plus fork. You cannot have both installed at the same time. To switch back to vanilla LIBERO, uninstall the fork and reinstall with `pip install -e ".[libero]"`.

Installation

System dependencies (Linux only)

sudo apt install libexpat1 libfontconfig1-dev libmagickwand-dev

Python package

pip install -e ".[libero]" "robosuite==1.4.1" bddl easydict mujoco wand scikit-image gym
git clone https://github.com/sylvestf/LIBERO-plus.git
cd LIBERO-plus && pip install --no-deps -e .
pip uninstall -y hf-libero  # so `import libero` resolves to the fork

LIBERO-plus is installed from its GitHub fork rather than a pyproject extra — the fork ships as a namespace package that pip can’t handle, so it must be cloned and added to PYTHONPATH. See docker/Dockerfile.benchmark.libero_plus for the canonical install. MuJoCo is required, so only Linux is supported.

Set the MuJoCo rendering backend before running evaluation:

Copied

export MUJOCO_GL=egl   # headless / HPC / cloud

Download LIBERO-plus assets

LIBERO-plus ships its extended asset pack separately. Download assets.zip from the Hugging Face dataset and extract it into the LIBERO-plus package directory:

# After installing the package, find where it was installed:
python -c "import libero; print(libero.__file__)"
# Then extract assets.zip into <package_root>/libero/assets/

Evaluation

Default evaluation (recommended)

Evaluate across the four standard suites (10 episodes per task):

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
  --eval.batch_size=1 \
  --eval.n_episodes=10 \
  --env.max_parallel_tasks=1

Single-suite evaluation

Evaluate on one LIBERO-plus suite:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial \
  --eval.batch_size=1 \
  --eval.n_episodes=10

• --env.task picks the suite (libero_spatial, libero_object, etc.).
• --env.task_ids restricts to specific task indices ([0], [1,2,3], etc.). Omit to run all tasks in the suite.
• --eval.batch_size controls how many environments run in parallel.
• --eval.n_episodes sets how many episodes to run per task.

Multi-suite evaluation

Benchmark a policy across multiple suites at once by passing a comma-separated list:

lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object \
  --eval.batch_size=1 \
  --eval.n_episodes=10

Control mode

LIBERO-plus supports two control modes — relative (default) and absolute. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:

--env.control_mode=relative   # or "absolute"

Policy inputs and outputs

Observations:

• observation.state — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
• observation.images.image — main camera view (agentview_image), HWC uint8
• observation.images.image2 — wrist camera view (robot0_eye_in_hand_image), HWC uint8

Actions:

• Continuous control in Box(-1, 1, shape=(7,)) — 6D end-effector delta + 1D gripper

Recommended evaluation episodes

For reproducible benchmarking, use 10 episodes per task across all four standard suites (Spatial, Object, Goal, Long). This gives 400 total episodes and matches the protocol used for published results.

Training

Dataset

A LeRobot-format training dataset for LIBERO-plus is available at:

• lerobot/libero_plus

Example training command

lerobot-train \
    --policy.type=smolvla \
    --policy.repo_id=${HF_USER}/smolvla_libero_plus \
    --policy.load_vlm_weights=true \
    --dataset.repo_id=lerobot/libero_plus \
    --env.type=libero_plus \
    --env.task=libero_spatial \
    --output_dir=./outputs/ \
    --steps=100000 \
    --batch_size=4 \
    --eval.batch_size=1 \
    --eval.n_episodes=1 \
    --env_eval_freq=1000

Relationship to LIBERO

LIBERO-plus is a drop-in extension of LIBERO:

• Same Python gym interface (LiberoEnv, LiberoProcessorStep)
• Same camera names and observation/action format
• Same task suite names
• Installs under the same libero Python package name (different GitHub repo)

To use the original LIBERO benchmark, see LIBERO and use --env.type=libero.