# LookStep

[English](README.md)

## 模型信息


| 字段                | 值                                 |
| ----------------- | --------------------------------- |
| Base model        | `Qwen/Qwen3-VL-8B-Instruct`       |
| Architecture      | `Qwen3VLForConditionalGeneration` |
| Model type        | `qwen3_vl`                        |
| 参数量               | 8,767,123,696                     |
| Checkpoint 格式     | safetensors，4 个分片，750 个 tensors   |
| Index 记录的权重字节数    | 17,534,247,392 bytes              |
| 微调方式              | 全参数 SFT（`tuner_type=full`）        |
| 最终 optimizer step | 18,888                            |
| 训练 epoch          | 1.0                               |
| 训练精度              | BF16                              |
| 训练最大长度            | 8,192 tokens                      |
| 输入                | 导航指令与前视 RGB observations          |
| 输出                | LookStep 结构化状态、候选后果、记忆决策和动作       |


在线 policy 接收 instruction、最多 6 帧长期事件记忆、最多 2 帧 recent observations 和
当前 RGB，并生成：

```xml
<progress>...</progress>
<event>...</event>
<memory_write>keep|drop</memory_write>
<memory_role>...</memory_role>
<outcomes>
  <move_forward>...</move_forward>
  <turn_left>...</turn_left>
  <turn_right>...</turn_right>
  <stop>...</stop>
</outcomes>
<action>MOVE_FORWARD|TURN_LEFT|TURN_RIGHT|STOP</action>
```


## 训练流程


| 超参数                         | 值                     |
| --------------------------- | --------------------- |
| GPUs                        | 8 × NVIDIA A100 80 GB |
| Epochs                      | 1                     |
| Per-device train batch size | 2                     |
| Gradient accumulation       | 8                     |
| Global batch size           | 128                   |
| Optimizer steps             | 18,888                |
| Optimizer                   | `adamw_torch_fused`   |
| Learning rate               | `2e-5`                |
| Scheduler                   | cosine                |
| Warmup ratio                | 0.03                  |
| Weight decay                | 0.01                  |
| Adam betas / epsilon        | 0.9, 0.95 / `1e-8`    |
| Max gradient norm           | 1.0                   |
| Distributed training        | DeepSpeed ZeRO-2      |
| Vision encoder              | frozen                |
| Visual aligner              | frozen                |
| LLM                         | trainable             |
| Model/data seeds            | 42 / 42               |


## 使用 LookStep 复现

创建固定环境并检查下载模型：

```bash
conda env create -f LookStep/simulation/environment.yml
conda activate lookstep-simulation

MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
bash LookStep/reproduce_paper.sh check-sim
```

先运行两个 episodes 的 smoke test，再运行完整主实验：

```bash
MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
bash LookStep/reproduce_paper.sh smoke-r2r

MODEL_PATH=/path/to/downloaded/checkpoint-18888 \
PROCESSOR_PATH=/path/to/Qwen3-VL-8B-Instruct \
DATA_ROOT=/path/to/data \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
bash LookStep/reproduce_paper.sh eval-all

bash LookStep/reproduce_paper.sh verify
```

## 引用

```
@inproceedings{
lookstep,
title={LookStep: Efficient Vision-Language Navigation with Linguistic Foresight and Event Driven Memory},
author={Kun-Yang Yu, Yingzhe Li, Hongyu Xu, Shi-Yu Tian, Zhi Zhou, Yang Chen, Ming Yang, Sheng Wang, Qing Yu, Lan-Zhe Guo, Yu-Feng Li},
booktitle={The 2026 Conference on Empirical Methods in Natural Language Processing},
year={2026}
}
```

如果有任何问题，请邮件联系yuky@lamda.nju.edu.cn (Kun-Yang Yu)