Gemma-4-E2B-it with calibrated NVFP4 KV-cache scales

google/gemma-4-E2B-it (weights unchanged, bf16) plus per-layer k_scale/v_scale for a 4-bit float KV cache, produced with llm-compressor 0.12 / compressed-tensors 0.17.1 (kv_cache_scheme: {num_bits: 4, type: float, strategy: tensor, observer: memoryless_minmax}, 256 HuggingFaceH4/ultrachat_200k samples at seq 2048; recipe in recipe.yaml).

Purpose: a checkpoint that vLLM can serve with --kv-cache-dtype nvfp4 on consumer Blackwell (RTX 5090 / GB10) via the FlashInfer FA2 path of vllm-project/vllm#46329, and the model behind vLLM's tests/v1/e2e/general/test_gemma4_nvfp4_kv_sm12x.py. Public "NVFP4" Gemma-4 checkpoints carry no KV scales and produce garbage with an NVFP4 KV cache; this one does not.

vllm serve jethachan/gemma-4-E2B-it-NVFP4KV-calib --kv-cache-dtype nvfp4 --language-model-only

Repro, probes and results: https://github.com/jethac/vllm-gemma4-nvfp4-kv-repro

Downloads last month
22
Safetensors
Model size
6B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for jethachan/gemma-4-E2B-it-NVFP4KV-calib

Quantized
(474)
this model