Voxtral Realtime NVFP4 for vLLM

Quantization fact Configuration
Model Complete Voxtral Mini 4B Realtime 2602 serving checkpoint
Quantized weights NVFP4: gate, up and down projections in all 26 decoder MLPs (78 linears)
Protected weights BF16: encoder, attention projections, embeddings, normalization and output head
Activations BF16 at every token position β€” W4A16
KV-cache storage FP8 E4M3 by default
Serving runtime vLLM 0.33.0, Marlin, single GPU
GPU target SM120 with sufficient VRAM, including RTX 50-series; benchmarked on RTX 5070 Ti 16 GiB
Aggregate quality DE +0.156 / EN βˆ’0.158 percentage points WER on the known 2,000-recording evaluation

35% lower model-loading GPU allocation, 31–43% more transcription throughput, and concurrent realtime streams on an RTX 5070 Ti.

Author: Dr. Lucas Spohn.
Attribution: Idana AG, Freiburg i. Br., Germany.
Original model and pretraining: Mistral AI.
Native Transformers alternative: NVFP4 MLP delta and sequential loader.

At a glance

Measured on one RTX 5070 Ti / SM120, with matched vLLM/Triton settings and the same GPU budget:

Measurement BF16 weights + BF16 KV NVFP4 weights + FP8 KV
GPU allocation while loading the model 8.43 GiB 5.52 GiB, about 35% less
Throughput, 1 / 2 / 4 / 8 clients 1.00x 1.312 / 1.348 / 1.418 / 1.426x
German WER, 1,000 recordings 4.987% 5.143%, +0.156 percentage points
English WER, 1,000 recordings 5.165% 5.006%, βˆ’0.158 percentage points

The loading-memory reduction leaves more space for KV cache. vLLM uses the available cache budget, so total running-server VRAM can be similar despite the smaller model allocation. With a single 2-GiB pool for other services and headroom, the tested server used approximately 13.4 GiB of the 16-GiB card.

Quality uses the same 2,000 previously evaluated FLEURS recordings, 42,648 reference words for each variant. It is not a new independent quality confirmation. Nineteen of the 34 fewer English errors come from one previously empty BF16 output. Full language-specific intervals and errors remain in EVALUATION.md.

Concurrent streams

Tested workload Result with NVFP4 + FP8 KV
8 simultaneous 150–165-second streams All complete; maximum finalization delay 1.058 s
32 simultaneous 10–16-second streams, three cohorts 96/96 complete; p95 0.599 s, maximum 0.612 s
Unpaced fixed-32-clip throughput at 32 clients, six runs Median 48.56 audio seconds per wall second
FP8 KV's additional throughput over NVFP4 + BF16 KV at 32 clients 5.3%, paired 95% interval 2.9–7.6%

Use eight streams as the tested starting configuration for longer inputs. Long 16-stream requests finished well beyond the two-second finalization point; long 24/32-stream tests timed out. Short-cohort throughput does not establish continuous capacity for 32 long conversations. Levels 9–15 remain unmeasured. Finalization is relative to stream start plus audio duration, including sender pacing lateness; it is not acoustic word lag.

Quick start

Tested: Linux, Python 3.12, NVIDIA RTX 5070 Ti (SM120), CUDA 13 and vLLM 0.33.0. Other SM120 cards with sufficient VRAM are expected to be compatible; their performance has not been measured here. SM120 identifies the GPU architecture, not a specific card: it includes RTX 5080/5090 as well as RTX 5070 Ti. Blackwell B200/GB200 accelerators use SM100 and B300/GB300 use SM103; these are not enabled by this SM120-only plugin. RTX PRO Blackwell models with SM120 belong to the supported architecture family, with performance still to be validated. The plugin currently supports a single GPU and rejects other GPU architectures. A custom quantization plugin is included; install it before loading this checkpoint. The complete model is included, so no separate base checkpoint is needed. The pinned kernel dependency may download on first start.

With uv:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install huggingface_hub==2.4.0
hf download lspohn/Voxtral-Mini-4B-Realtime-2602-NVFP4-vLLM-SM120 \
  --revision v0.2.0 --local-dir voxtral-nvfp4-vllm
cd voxtral-nvfp4-vllm
uv pip sync requirements.lock
uv pip install --no-deps runtime/idana_voxtral_vllm_nvfp4-0.2.0-py3-none-any.whl
voxtral-nvfp4-serve . --streams 8

The server listens at 127.0.0.1:8000, with model name voxtral and WebSocket endpoint ws://127.0.0.1:8000/v1/realtime. Startup includes kernel preparation and graph capture. Transcribe a mono PCM16 WAV at 16 kHz:

voxtral-nvfp4-transcribe audio.wav
voxtral-nvfp4-transcribe first.wav second.wav third.wav

Multiple files use independent concurrent connections. Audio is paced in realtime by default; --unpaced sends it as fast as possible.

The launcher selects Marlin, FP8 E4M3 KV, context 8192, batch-token limit 128 and FULL_DECODE_ONLY graphs. Its default --reserve-gib 2 counts other services and their headroom together, once. It logs owned-process CPU, RAM and VRAM, pauses at 80Β°C for at least 15 minutes, and resumes at or below 75Β°C. Use --gpu, --port, --out and --dry-run to inspect or configure a launch. The 8192-position configuration has a finite stream-length ceiling; the long tests cover 150–165 seconds, not unlimited-duration sessions.

What is quantized?

Exactly 78 decoder MLP projections: gate, up and down in all 26 decoder layers. The encoder, attention projections, embeddings, normalization and output head keep BF16 weights. Separate gate/up global scales are preserved. The original FP4 codes and scale values are retained, with runtime repacking for Marlin. Activations stay BF16 at all token positions, including prefill; this is W4A16, not end-to-end W4A4 inference. FP8 is the KV storage format; the selected Triton attention path computes with BF16 values.

Calibration used activation statistics from 64 general public FLEURS clips. This is post-training quantization without transcript-label training or adapters. The weights descend from the linked native-runtime release; this release adds the full serving export, vLLM integration and parallel-serving evaluation.

Native INT8 attention was also evaluated, but was slower than the FP8-cache baseline in the tested operator pilot.

Results, code and attribution

The original native Transformers implementation remains available, including its own sequential-runtime measurements.

Downloads last month
40
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for lspohn/Voxtral-Mini-4B-Realtime-2602-NVFP4-vLLM-SM120