README_ZH.md

LookStep

English

模型信息

字段	值
Base model	Qwen/Qwen3-VL-8B-Instruct
Architecture	Qwen3VLForConditionalGeneration
Model type	qwen3_vl
参数量	8,767,123,696
Checkpoint 格式	safetensors，4 个分片，750 个 tensors
Index 记录的权重字节数	17,534,247,392 bytes
微调方式	全参数 SFT（tuner_type=full）
最终 optimizer step	18,888
训练 epoch	1.0
训练精度	BF16
训练最大长度	8,192 tokens
输入	导航指令与前视 RGB observations
输出	LookStep 结构化状态、候选后果、记忆决策和动作

在线 policy 接收 instruction、最多 6 帧长期事件记忆、最多 2 帧 recent observations 和 当前 RGB，并生成：

<progress>...</progress>
<event>...</event>
<memory_write>keep|drop</memory_write>
<memory_role>...</memory_role>
<outcomes>
  <move_forward>...</move_forward>
  <turn_left>...</turn_left>
  <turn_right>...</turn_right>
  <stop>...</stop>
</outcomes>
<action>MOVE_FORWARD|TURN_LEFT|TURN_RIGHT|STOP</action>

训练流程

超参数	值
GPUs	8 × NVIDIA A100 80 GB
Epochs	1
Per-device train batch size	2
Gradient accumulation	8
Global batch size	128
Optimizer steps	18,888
Optimizer	adamw_torch_fused
Learning rate	2e-5
Scheduler	cosine
Warmup ratio	0.03
Weight decay	0.01
Adam betas / epsilon	0.9, 0.95 / 1e-8
Max gradient norm	1.0
Distributed training	DeepSpeed ZeRO-2
Vision encoder	frozen
Visual aligner	frozen
LLM	trainable
Model/data seeds	42 / 42

使用 LookStep 复现

创建固定环境并检查下载模型：

conda env create -f LookStep/simulation/environment.yml
conda activate lookstep-simulation

MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
bash LookStep/reproduce_paper.sh check-sim

先运行两个 episodes 的 smoke test，再运行完整主实验：

MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
bash LookStep/reproduce_paper.sh smoke-r2r

MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
bash LookStep/reproduce_paper.sh eval-all

bash LookStep/reproduce_paper.sh verify

引用

@inproceedings{
lookstep,
title={LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory},
author={Kun-Yang Yu, Yingzhe Li, Hongyu Xu, Shi-Yu Tian, Zhi Zhou, Yang Chen, Ming Yang, Sheng Wang, Qing Yu, Lan-Zhe Guo, Yu-Feng Li},
booktitle={The 2026 Conference on Empirical Methods in Natural Language Processing},
year={2026}
}

如果有任何问题，请邮件联系yuky@lamda.nju.edu.cn (Kun-Yang Yu)