Voxtral Realtime NVFP4 for vLLM
| Quantization fact | Configuration |
|---|---|
| Model | Complete Voxtral Mini 4B Realtime 2602 serving checkpoint |
| Quantized weights | NVFP4: gate, up and down projections in all 26 decoder MLPs (78 linears) |
| Protected weights | BF16: encoder, attention projections, embeddings, normalization and output head |
| Activations | BF16 at every token position β W4A16 |
| KV-cache storage | FP8 E4M3 by default |
| Serving runtime | vLLM 0.33.0, Marlin, single GPU |
| GPU target | SM120 with sufficient VRAM, including RTX 50-series; benchmarked on RTX 5070 Ti 16 GiB |
| Aggregate quality | DE +0.156 / EN β0.158 percentage points WER on the known 2,000-recording evaluation |
35% lower model-loading GPU allocation, 31β43% more transcription throughput, and concurrent realtime streams on an RTX 5070 Ti.
Author: Dr. Lucas Spohn.
Attribution: Idana AG, Freiburg i. Br., Germany.
Original model and pretraining: Mistral AI.
Native Transformers alternative: NVFP4 MLP delta and sequential loader.
At a glance
Measured on one RTX 5070 Ti / SM120, with matched vLLM/Triton settings and the same GPU budget:
| Measurement | BF16 weights + BF16 KV | NVFP4 weights + FP8 KV |
|---|---|---|
| GPU allocation while loading the model | 8.43 GiB | 5.52 GiB, about 35% less |
| Throughput, 1 / 2 / 4 / 8 clients | 1.00x | 1.312 / 1.348 / 1.418 / 1.426x |
| German WER, 1,000 recordings | 4.987% | 5.143%, +0.156 percentage points |
| English WER, 1,000 recordings | 5.165% | 5.006%, β0.158 percentage points |
The loading-memory reduction leaves more space for KV cache. vLLM uses the available cache budget, so total running-server VRAM can be similar despite the smaller model allocation. With a single 2-GiB pool for other services and headroom, the tested server used approximately 13.4 GiB of the 16-GiB card.
Quality uses the same 2,000 previously evaluated FLEURS recordings, 42,648 reference words for each variant. It is not a new independent quality confirmation. Nineteen of the 34 fewer English errors come from one previously empty BF16 output. Full language-specific intervals and errors remain in EVALUATION.md.
Concurrent streams
| Tested workload | Result with NVFP4 + FP8 KV |
|---|---|
| 8 simultaneous 150β165-second streams | All complete; maximum finalization delay 1.058 s |
| 32 simultaneous 10β16-second streams, three cohorts | 96/96 complete; p95 0.599 s, maximum 0.612 s |
| Unpaced fixed-32-clip throughput at 32 clients, six runs | Median 48.56 audio seconds per wall second |
| FP8 KV's additional throughput over NVFP4 + BF16 KV at 32 clients | 5.3%, paired 95% interval 2.9β7.6% |
Use eight streams as the tested starting configuration for longer inputs. Long 16-stream requests finished well beyond the two-second finalization point; long 24/32-stream tests timed out. Short-cohort throughput does not establish continuous capacity for 32 long conversations. Levels 9β15 remain unmeasured. Finalization is relative to stream start plus audio duration, including sender pacing lateness; it is not acoustic word lag.
Quick start
Tested: Linux, Python 3.12, NVIDIA RTX 5070 Ti (SM120), CUDA 13 and vLLM 0.33.0. Other SM120 cards with sufficient VRAM are expected to be compatible; their performance has not been measured here. SM120 identifies the GPU architecture, not a specific card: it includes RTX 5080/5090 as well as RTX 5070 Ti. Blackwell B200/GB200 accelerators use SM100 and B300/GB300 use SM103; these are not enabled by this SM120-only plugin. RTX PRO Blackwell models with SM120 belong to the supported architecture family, with performance still to be validated. The plugin currently supports a single GPU and rejects other GPU architectures. A custom quantization plugin is included; install it before loading this checkpoint. The complete model is included, so no separate base checkpoint is needed. The pinned kernel dependency may download on first start.
With uv:
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install huggingface_hub==2.4.0
hf download lspohn/Voxtral-Mini-4B-Realtime-2602-NVFP4-vLLM-SM120 \
--revision v0.2.0 --local-dir voxtral-nvfp4-vllm
cd voxtral-nvfp4-vllm
uv pip sync requirements.lock
uv pip install --no-deps runtime/idana_voxtral_vllm_nvfp4-0.2.0-py3-none-any.whl
voxtral-nvfp4-serve . --streams 8
The server listens at 127.0.0.1:8000, with model name voxtral and WebSocket
endpoint ws://127.0.0.1:8000/v1/realtime. Startup includes kernel preparation
and graph capture. Transcribe a mono PCM16 WAV at 16 kHz:
voxtral-nvfp4-transcribe audio.wav
voxtral-nvfp4-transcribe first.wav second.wav third.wav
Multiple files use independent concurrent connections. Audio is paced in
realtime by default; --unpaced sends it as fast as possible.
The launcher selects Marlin, FP8 E4M3 KV, context 8192, batch-token limit 128
and FULL_DECODE_ONLY graphs. Its default --reserve-gib 2 counts other
services and their headroom together, once. It logs owned-process CPU, RAM and
VRAM, pauses at 80Β°C for at least 15 minutes, and resumes at or below 75Β°C.
Use --gpu, --port, --out and --dry-run to inspect or configure a launch.
The 8192-position configuration has a finite stream-length ceiling; the long
tests cover 150β165 seconds, not unlimited-duration sessions.
What is quantized?
Exactly 78 decoder MLP projections: gate, up and down in all 26 decoder layers. The encoder, attention projections, embeddings, normalization and output head keep BF16 weights. Separate gate/up global scales are preserved. The original FP4 codes and scale values are retained, with runtime repacking for Marlin. Activations stay BF16 at all token positions, including prefill; this is W4A16, not end-to-end W4A4 inference. FP8 is the KV storage format; the selected Triton attention path computes with BF16 values.
Calibration used activation statistics from 64 general public FLEURS clips. This is post-training quantization without transcript-label training or adapters. The weights descend from the linked native-runtime release; this release adds the full serving export, vLLM integration and parallel-serving evaluation.
Native INT8 attention was also evaluated, but was slower than the FP8-cache baseline in the tested operator pilot.
Results, code and attribution
- EVALUATION.md: complete comparison, intervals, resources and limits.
- results.json: aggregate metrics, six-pair ratios and all capacity outcomes.
- runtime/: installable plugin wheel and complete corresponding source.
- evaluation/: public fixture indices, reconstruction and benchmark tools.
- checksums.json: checkpoint and release-file integrity.
- NOTICE, LICENSE, CITATION.cff.
The original native Transformers implementation remains available, including its own sequential-runtime measurements.
- Downloads last month
- 40
Model tree for lspohn/Voxtral-Mini-4B-Realtime-2602-NVFP4-vLLM-SM120
Base model
mistralai/Ministral-3-3B-Base-2512