PRIMO R1: Process Reasoning Induced Monitoring

PRIMO R1 (Process Reasoning Induced Monitoring) is a 7B video multimodal large language model (MLLM) framework designed for accurate process supervision in long-horizon robotic manipulation. It was introduced in the paper From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation.

This is the final checkpoint: a Qwen2.5-VL-7B-Instruct base taken through an SFT cold start and then GRPO reinforcement learning. Use this model unless you specifically want the ablation.

Model Description

Current video MLLMs often function as passive "Observers" that recognize ongoing events rather than evaluating the current state relative to the final task goal. PRIMO R1 transforms these models into active "Critics" by:

Reinforcement Learning: Leveraging outcome-based RL to incentivize explicit Chain-of-Thought (CoT) generation for progress estimation.
Temporal Anchoring: Constructing a structured temporal input that explicitly anchors the video sequence between initial and current state images.
Process Reasoning: Focusing on evaluating the current state against the intended task goal to detect failures and track progress.

Resources


Code	10-OASIS-01/PRIMO-R1
Collection	PRIMO R1
Paper	arXiv 2603.15600 · project page
Stage-1 ablation	PRIMO-COT-SFT-7B
Benchmark	primo-bench-json
Training data	primo-sft-json · primo-rl-json
Videos	primo-video-media

Download

The main branch holds inference files only (16.6 GB). DeepSpeed resume state from RL step 2500 lives on a separate branch, so you do not pay ~100 GB for it by accident.

from huggingface_hub import snapshot_download

# Inference: 16.6 GB
snapshot_download("LeonOverload/PRIMO-R1-7B", local_dir="models/PRIMO-R1-7B")

# Resume RL training from step 2500: ~116 GB
snapshot_download("LeonOverload/PRIMO-R1-7B", revision="training-state")

Or from the CLI:

hf download LeonOverload/PRIMO-R1-7B --local-dir models/PRIMO-R1-7B

If you are on a revision where the optimizer state is still on main, exclude it explicitly. Note that --exclude is ignored if you also pass filenames positionally:

hf download LeonOverload/PRIMO-R1-7B \
    --exclude "global_step*" --exclude "rng_state*" \
    --exclude "scheduler.pt" --exclude "latest" \
    --local-dir models/PRIMO-R1-7B

Setup

The environment matters more than usual here. Qwen2.5-VL support shifted between transformers releases, and installing a PyPI transformers over the pinned tree is the most common cause of shape and processor errors:

git clone https://github.com/10-OASIS-01/PRIMO-R1 && cd PRIMO-R1
conda create -n primo-r1 python=3.11 && conda activate primo-r1
bash setup.sh

setup.sh pins vllm==0.7.2, trl==0.16.0, and installs the vendored transformers-main/ tree last so nothing replaces it.

Input format — read this before running anything

PRIMO R1 is trained on an interleaved input that anchors the clip between two still frames. The content list must be in exactly this order:

image (initial frame)  →  video (the clip)  →  image (current frame)  →  text (question)

Feeding a bare video clip will still produce output, but it degrades quality silently — the paper's Table 4 ablation measures this directly, and a current-state image alone raises average MAE from 15.52 to 59.50. Keep the order and keep both anchor frames.

The prompt template is equally load-bearing: the answer extractor is a regex over <answer>...</answer>, and the <think> block is required to contain <planning>, <observation>, and <reasoning> subsections in that order. Changing the template without changing the extractor collapses scores rather than raising an error.

Usage

This example is transcribed from src/eval/eval_interleave.py, the harness that produced the published numbers.

The prompts and the frame extraction are imported, not pasted. Both live in the repo cloned during setup: src/primo_prompts.py holds every prompt constant and src/primo_video_utils.py holds the frame helpers. They are the same objects the eval harness uses, so an example that imports them cannot drift out of sync with the checkpoint. setup.sh puts src/ on PYTHONPATH; from elsewhere, add it yourself:

import sys
sys.path.insert(0, "/path/to/PRIMO-R1/src")

import torch
from transformers import AutoProcessor, AutoTokenizer
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info

# The single source of truth for the prompt format and the anchor frames.
from primo_prompts import SYSTEM_PROMPT, build_question
from primo_video_utils import extract_frames_on_demand

MODEL_PATH = "models/PRIMO-R1-7B"   # or "LeonOverload/PRIMO-R1-7B"
video_path = "path/to/your/episode.mp4"
question = "What is the completion percentage of the task in the video?"
problem_type = "regression"

# (initial state, current state) as PIL images. LRU-cached, so calling this
# again for the same video is free.
init_img, current_img = extract_frames_on_demand(video_path)

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {
        "role": "user",
        "content": [
            {"type": "image", "image": init_img},                      # 1. initial state
            {"type": "video", "video": video_path, "nframes": 22},     # 2. the clip
            {"type": "image", "image": current_img},                   # 3. current state
            # QUESTION_TEMPLATE.format(...) + TYPE_TEMPLATE[problem_type]
            {"type": "text", "text": build_question(question, problem_type)},
        ],
    },
]

llm = LLM(
    model=MODEL_PATH,
    tensor_parallel_size=torch.cuda.device_count(),
    max_model_len=16384,
    gpu_memory_utilization=0.8,
    limit_mm_per_prompt={"image": 3, "video": 1},   # 2 anchor frames + 1 video
)

# top_p must stay this low. Larger values produce garbled output on this model.
sampling_params = SamplingParams(temperature=0.1, top_p=0.001, max_tokens=4096)

processor = AutoProcessor.from_pretrained(MODEL_PATH)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
tokenizer.padding_side = "left"
processor.tokenizer = tokenizer

prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)

mm_data = {"video": video_inputs[0]}
if image_inputs:
    mm_data["image"] = image_inputs

outputs = llm.generate(
    [{
        "prompt": prompt,
        "multi_modal_data": mm_data,
        "mm_processor_kwargs": {k: v[0] for k, v in video_kwargs.items()},
    }],
    sampling_params=sampling_params,
)
print(outputs[0].outputs[0].text)

Parsing the output

import re

def extract_answer(text):
    m = re.search(r"<answer>\s*(.*?)\s*</answer>", text, re.DOTALL)
    return m.group(1).strip() if m else ""

def extract_think(text):
    m = re.search(r"<think>\s*(.*?)\s*</think>", text, re.DOTALL)
    return m.group(1).strip() if m else ""

For regression and numerical questions the answer is a progress percentage on a 0–100 scale (42.5, not 0.425). A trailing % is tolerated by the reference parser, which divides by 100 when it sees one.

Question types

regression and numerical (progress estimation), multiple choice, boolean (failure detection), free-form, OCR. build_question puts the type in both places — inside QUESTION_TEMPLATE and as the TYPE_TEMPLATE hint appended after it — because the model was trained with both present.

The prompt constants

If you need to inspect or extend them rather than just call build_question:

from primo_prompts import SYSTEM_PROMPT, QUESTION_TEMPLATE, TYPE_TEMPLATE

print(QUESTION_TEMPLATE.format(Question="...", question_type="regression"))
print(sorted(TYPE_TEMPLATE))
# ['OCR', 'boolean', 'free-form', 'multiple choice', 'numerical', 'regression']

primo_prompts.py also carries the baseline and ablation prompts under *_BASELINE / *_PARSER_ONLY / *_QUESTION_ONLY names. Those are for reproducing the paper's comparison rows, not for this checkpoint — it was trained on QUESTION_TEMPLATE and expects it.

A note on frame counts

The published results were produced with the per-video frame cap at 22, so the launcher's --nframes 32 effectively sampled 22. nframes=22 above reproduces that; primo_video_utils.choose_nframes is what applies the cap in the harness (MAX_NFRAMES, overridable via INTERLEAVE_MAX_NFRAMES). Training capped videos at 16 frames; eval samples more at higher resolution.

Performance

Progress estimation averaged over four environments (paper Table 1):

Model	Avg MRA ↑	Avg MAE ↓
GPT-4o	79.33	20.67
GPT-5 mini	75.38	23.96
Qwen2.5-VL-72B	73.80	23.80
Qwen2.5-VL-7B (base)	67.79	29.99
InternVL 3.5 8B	71.74	28.09
ProgressLM	78.32	20.87
VLAC	74.90	25.10
PRIMO R1 (7B)	82.90	15.52

Both training stages are needed (paper Table 2, MRA↑):

Model	ID avg	OOD avg	Overall
Qwen2.5-VL-7B (base)	70.38	65.26	67.46
SFT only	81.46	77.77	79.35
RL only	81.71	72.97	76.72
PRIMO R1 (SFT+RL)	88.47	82.90	85.28

RL without SFT underperforms because the model struggles to discover the output format from scratch; SFT alone generalizes poorly out of domain.

Zero-shot failure detection on RoboFail (paper Table 3): 67.0%, matching Gemini 2.0 Flash and above GPT-4o at 63.0.

To reproduce these numbers, see primo-bench-json, which documents the full evaluation path. One caveat when comparing against other work: the repo's harnesses score regression with three different formulas, and each output file records which one it used in its regression_metric field. The published numbers above come from linear_relative_accuracy.

Citations

If you find our work helpful for your research, please consider citing our work.

@misc{liu2026passiveobserveractivecritic,
      title={From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation},
      author={Yibin Liu and Yaxing Lyu and Daqi Gao and Zhixuan Liang and Weiliang Tang and Shilong Mu and Xiaokang Yang and Yao Mu},
      year={2026},
      eprint={2603.15600},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.15600},
}