---
license: apache-2.0
language:
- en
base_model:
- Efficient-Large-Model/Fast_dVLM_3B
tags:
- diffusion
- vlm
- block-diffusion
- parallel-decoding
- quantization
- fp8
- smoothquant
---

# Fast-dVLM (3B) — W8A8 FP8 Quantized

[[Paper](https://arxiv.org/abs/2604.06832)] [[Project Page](https://nvlabs.github.io/Fast-dLLM/fast_dvlm/)] [[Code](https://github.com/NVlabs/Fast-dLLM)] [[BF16 Base](https://huggingface.co/Efficient-Large-Model/Fast_dVLM_3B)]

## Introduction

This repository hosts the **W8A8 FP8 quantized** version of [`Fast_dVLM_3B`](https://huggingface.co/Efficient-Large-Model/Fast_dVLM_3B), produced via SmoothQuant offline calibration.

Both the language tower **and** the vision encoder are quantized to **FP8 (E4M3)** weights with per-channel static scales and per-token dynamic FP8 activations. Combined with SGLang block-diffusion serving, this checkpoint reaches **350 TPS (6.18× over the AR baseline)** while keeping MMMU-Pro-V within 0.3 of the BF16 result.

### Key Highlights

- **6.18× Speedup** over the Qwen2.5-VL-3B AR baseline (350 TPS vs 56.7 TPS).
- **Near-Lossless Quality**: 23.8 vs 24.1 MMMU-Pro-V vs the BF16 block-diffusion model.
- **Full-Model FP8**: Language + vision both quantized; only norms, embeddings, and `lm_head` remain in BF16.

---

## Model Overview

| Property | Value |
|---|---|
| **Type** | Block Diffusion Vision-Language Model (FP8 quantized) |
| **Base Model** | [`Efficient-Large-Model/Fast_dVLM_3B`](https://huggingface.co/Efficient-Large-Model/Fast_dVLM_3B) |
| **Quantization** | SmoothQuant W8A8 FP8 (E4M3), per-channel static weight / per-token dynamic activation |
| **Calibration** | 512 samples × 1024 tokens on C4, SmoothQuant α=0.5 |
| **Text Layers** | 36 |
| **Vision Depth** | 32 |
| **Text Hidden Size** | 2048 |
| **Attention Heads** | 16 (Q), 2 (KV, GQA) |
| **Block Diffusion Size** | 32 |

**Hardware requirement**: NVIDIA GPU with **SM89+ (Compute Capability ≥ 8.9)** — RTX 4090, L40, H100, H200. FP8 tensor cores are not available on A100 (SM80) or older.

---

## Quickstart (SGLang)

Load this checkpoint with the customized SGLang shipped in the [Fast-dLLM repo](https://github.com/NVlabs/Fast-dLLM):

```bash
# Install the customized SGLang (one-time)
git clone https://github.com/NVlabs/Fast-dLLM
cd Fast-dLLM/fast_dvlm/sglang/python
pip install -e .

# Run the chatbot with FP8 quantization
cd ../..
python run_chatbot_sglang.py \
    --algorithm spec \
    --model-path Sensen02/Fast_dVLM_3B_W8A8_FP8 \
    --quantization w8a8_fp8 \
    --prompt "Describe this image." \
    --image path/to/image.jpg
```

SGLang reads the `quantization_config` in `config.json` automatically:

```json
"quantization_config": {
    "quant_method": "w8a8_fp8",
    "is_dynamic": false,
    "ignore": []
}
```

The `ignore` list is empty because every linear layer — including the vision encoder — is FP8 quantized in this release.

---

## Benchmark Results

### Quality (MMMU-Pro-V)

| Variant | MMMU-Pro-V |
|---|---|
| Qwen2.5-VL-3B (AR baseline) | 26.3 |
| Fast-dVLM (MDM, τ=0.9) | 21.4 |
| Fast-dVLM (spec.) — BF16 | 24.1 |
| **Fast-dVLM (spec.) — W8A8 FP8 (this repo)** | **23.8** |

### Inference Acceleration

| Setting | MMMU-Pro-V | TPS | SpeedUp |
|---|---|---|---|
| AR baseline | 26.3 | 56.7 | 1.00x |
| Fast-dVLM (MDM, τ=0.9) | 21.4 | 82.2 | 1.45x |
| + Spec. decoding (linear) | 24.6 | 112.7 | 1.98x |
| + SGLang serving | 24.1 | 319.0 | 5.63x |
| **+ SmoothQuant-W8A8 (FP8)** | **23.8** | **350.3** | **6.18x** |

---

## Citation

If you use Fast-dVLM in your research, please cite:

```bibtex
@misc{wu2026fastdvlmefficientblockdiffusionvlm,
      title={Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM},
      author={Chengyue Wu and Shiyi Lan and Yonggan Fu and Sensen Gao and Jin Wang and Jincheng Yu and Jose M. Alvarez and Pavlo Molchanov and Ping Luo and Song Han and Ligeng Zhu and Enze Xie},
      year={2026},
      eprint={2604.06832},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.06832},
}
```

---

## License

Released under **Apache 2.0**, following the base Qwen2.5-VL license.