MarshallHD/zerank-2-reranker-NVFP4

zerank-2-reranker — NVFP4 (W4A4)

This repository contains an NVFP4 (4-bit) quantization of zeroentropy/zerank-2-reranker, prepared for NVIDIA Blackwell GPUs and validated on a DGX Spark (GB10). This card documents the quantization method, the measured quality differences against the bf16 original, and the serving configuration we validated. Numbers are reported as measured; limitations are listed at the end.

The motivation was serving memory. A reranker processes one short query–document pair per request, but served with common vLLM settings the bf16 model reserves tens of gigabytes of memory for capacity it never uses. Quantizing the weights and right-sizing the serving configuration brings the same model to roughly 5.8 GiB.

	bf16	NVFP4 (this repo)
Checkpoint size	8.06 GB	2.83 GB
Serving memory (validated configuration)	~14 GiB	~5.8 GiB
Throughput, GB10, 100-pair batches	657 pairs/s	1,305 pairs/s
MAP@1000, two MTEB tasks	baseline	−0.94% and −1.55% relative

Throughput was measured with a small fixed benchmark on a single machine and should be treated as indicative rather than a load test.

Quantization method

We used llm-compressor to apply NVFP4 W4A4 quantization, exported in compressed-tensors format. The quantization is selective rather than uniform.

Quantized: the 252 linear layers in the 36 transformer blocks (attention and MLP projections), which hold most of the parameter mass.

Kept in bf16, byte-identical to the original checkpoint:

• the token embeddings, which are tied to the output head;
• all 144 per-layer normalization tensors;
• the final normalization tensor.

The embedding protection is deliberate. zerank-2 scores a query–document pair by reading the logit of the Yes token, and because the embeddings are tied to the output head, they are the scoring path. Quantizing them would perturb every score directly. We verified after export that each protected tensor is byte-identical to the base model.

Activation scales were calibrated on 512 query–document reranking pairs formatted through the model's own chat template with add_generation_prompt=true, so calibration inputs match the token distributions the model sees when serving.

Evaluation

Both precisions were evaluated with MTEB 2.7.13 (MAP@1000), scored through a live vLLM /v1/rerank endpoint using the same harness.

Task	bf16	NVFP4	Δ absolute	Δ relative
AskUbuntuDupQuestions	0.65789	0.65172	−0.00617	−0.94%
SciDocsRR	0.87316	0.85961	−0.01355	−1.55%

These are metric-relative differences, not estimates of end-task answer quality. The degradation is small but consistent. Applications that are sensitive to small ranking-quality differences should use the bf16 original; this quantization trades that margin for lower memory use and higher throughput.

One serving behavior worth knowing: vLLM is not deterministic across server restarts. In our runs, a few documents per hundred shifted by up to about one logit point between cold starts (largest recorded: 1.62). Ranking metrics were unaffected in every case we measured, but applications that depend on numerically stable raw scores across restarts should verify this in their own setup.

Serving with vLLM

Validated with vLLM 0.25.1. Download this repository to a local directory and point MODEL_PATH at it so vLLM can load the included chat_template.jinja.

export MODEL_PATH=/path/to/zerank-2-reranker-nvfp4

vllm serve "$MODEL_PATH" \
  --dtype bfloat16 \
  --served-model-name zerank-2-nvfp4 \
  --host 127.0.0.1 \
  --port 8000 \
  --runner pooling \
  --convert classify \
  --hf-overrides '{"method":"no_post_processing","classifier_from_token":["Yes"],"num_labels":1,"use_sep_token":false}' \
  --chat-template "$MODEL_PATH/chat_template.jinja" \
  --default-chat-template-kwargs '{"add_generation_prompt":true}' \
  --quantization compressed-tensors \
  --linear-backend cutlass \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.05 \
  --max-num-seqs 128 \
  --no-enable-log-requests

Do not serve this model without its chat template. A server started without --chat-template and --default-chat-template-kwargs will pass health checks and return scores, but the scores will not discriminate between relevant and irrelevant documents. This is the most common misconfiguration for this model. Keep both flags in the serve command and include chat_template_kwargs in client requests.

Example request:

curl http://127.0.0.1:8000/v1/rerank \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "zerank-2-nvfp4",
    "query": "What is 2+2?",
    "documents": [
      "4",
      "The answer is definitely 1 million",
      "Paris is the capital of France."
    ],
    "use_activation": false,
    "chat_template_kwargs": {"add_generation_prompt": true}
  }'

With use_activation: false and the no_post_processing override, the endpoint returns the raw Yes-token logit. Higher scores indicate greater relevance.

Hardware and memory configurations

Native NVFP4 execution requires Blackwell-class CUTLASS kernel support (SM100 or newer; we tested SM121). Validation covered one DGX Spark GB10 with vLLM 0.25.1. Fallback behavior on earlier GPU generations was not tested.

Configuration	max-model-len / gpu-memory-utilization / max-num-seqs	Observed serving memory	Status
8k context	8192 / 0.05 / 128	~5.8 GiB	validated, recommended
32k context, low memory	32768 / 0.08 / 128	9.1–9.8 GiB	experimental
bf16 reference	32768 / 0.12 / 128	~14 GiB	reference

The 32k low-memory configuration started and served near-full-context requests, but its AskUbuntu result missed our pre-declared equivalence threshold against the validated configuration, so we do not recommend it without independent validation. Memory figures are process-level readings on GB10 unified memory; readings on discrete-memory systems may differ.

Limitations

• Ranking quality is measurably below bf16 on both benchmarks evaluated: −0.94% and −1.55% relative MAP@1000.
• Validation covered a single machine (DGX Spark GB10). Other Blackwell hardware, including x86 systems, is expected to work but was not tested.
• Evaluation covered two English MTEB tasks. Other languages and domains were not evaluated.
• Raw logits vary slightly across server restarts even when ranking metrics do not.
• The recommended configuration caps context at 8,192 tokens; the 32k low-memory configuration is experimental.

License and attribution

This model inherits the base model's CC BY-NC 4.0 license and is for non-commercial use only.

This is a quantized derivative of zeroentropy/zerank-2-reranker by ZeroEntropy. The changes are limited to NVFP4 quantization of the transformer linear layers; embeddings and normalization tensors are byte-identical to the base checkpoint. The quantization recipe is included in this repository (recipe.yaml).

Built with llm-compressor and served with vLLM.