Qwen3-Embedding-4B-W8A16 (Quadux)

This is a W8A16 (8-bit weights, 16-bit activations) quantization of Qwen/Qwen3-Embedding-4B, produced by Quadux IT GmbH with llm-compressor for direct serving via vLLM.

⚠️ Hardware: NVIDIA GPU (CUDA) only

This W8A16 quant relies on the Marlin kernel family, which is implemented for CUDA only. vLLM's CPU backend cannot load this model — it accepts only uint4 (AWQ / GPTQ W4A16) or compressed-tensors INT8 W8A8, not the uint8b128 format W8A16 produces. Tested with vLLM v0.20.0 — see issue #33970 for the kernel-level discussion. For CPU deployment use the AWQ W4A16 variant instead.

Overview

Property	Value
Model size on disk	5 GB (vs. ~8 GB BF16 original)
Format	compressed-tensors (W8A16 (8-bit weights, 16-bit activations))
Native embedding dimension	2560
Matryoshka dimensions	[128, 256, 384, 512, 768, 1024, 1536, 2048, 2560]
Max sequence length	32768 tokens
Languages	100+ (multilingual)
Target hardware	CUDA GPUs. Not for CPU.

Why use this quant

• 50% smaller on disk than the BF16 source (9 GB vs ~17 GB for 8B)
• Negligible quality loss for retrieval: cosine ≥ 0.998 vs BF16 source on 20 realistic German texts (mails, invoices, engineering data), Top-1-nearest-neighbour agreement 20/20.
• Production-ready on any CUDA GPU with compressed-tensors support (vLLM, SGLang). Not for vLLM CPU — see the AWQ W4A16 variant for that.

Quick start with vLLM

docker run --rm -p 1236:1236 \
    --gpus all \ \
    -v $(pwd)/models:/models:ro \
    vllm/vllm-openai:latest \ \
    --model QuaduxIT/Qwen3-Embedding-4B-W8A16 \
    --served-model-name qwen3-embedding \
    --port 1236 --host 0.0.0.0 \
    --trust-remote-code \
    --hf-overrides='{"is_matryoshka":true,"matryoshka_dimensions":[128,256,384,512,768,1024,1536,2048,3072,4096]}'

Then call the OpenAI-compatible endpoint:

curl -s -X POST http://localhost:1236/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding","input":"Sample text","dimensions":1024}'

The response is OpenAI-conformant; the server slices the native 2560-dim vector to your requested dimensions and re-normalises to L2 = 1.

Quantization details

• Algorithm: Round-to-Nearest with llm-compressor QuantizationModifier

• Scheme: W8A16

• Group size: 128 (per-channel within groups)

• Calibration: 128 samples from HuggingFaceH4/ultrachat_200k, max_seq_length=2048

• Layers excluded from quantization: lm_head, embed_tokens

• Output format: compressed-tensors (vLLM-native, no special flags needed)

• Tooling version: llm-compressor>=0.4.0,<0.7

• Recipe file: recipe.yaml is shipped with the model and documents the exact quantization configuration.

Quality vs. BF16 source

Verified on the Quadux internal fixture (20 realistic German texts spanning mails, invoices, engineering documentation, knowledge content, and questions):

Metric	Value
Cosine similarity vs BF16 source (median)	0.9992
Cosine similarity vs BF16 source (worst case)	0.9986
Maximum component drift	< 0.01
Top-1 nearest-neighbour agreement	20/20 (100%)

For retrieval the W8A16 vector is functionally indistinguishable from the BF16 reference: every test query lands on the exact same nearest neighbour as the original model would.

Hardware compatibility

Backend	Status
vLLM CUDA (Marlin / compressed-tensors)	✅ verified
vLLM CPU	❌ not supported (W8A16 / uint8b128 not in CPU kernels)
TensorRT-LLM	likely (untested by us)
SGLang	likely (compressed-tensors support)

For CPU deployment use the W4A16 (AWQ) variant instead.

Files

Qwen3-Embedding-4B-W8A16/
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── chat_template.jinja
├── recipe.yaml             # quantization recipe (reproducibility)
├── model.safetensors.index.json
├── model-00001-of-00002.safetensors
└── model-00002-of-00002.safetensors

License

This model and its base model are licensed under the Apache License 2.0. The Apache 2.0 license permits commercial and research use, modification, and redistribution, subject to the standard requirements: include the copyright notice, the license text, and a NOTICE of any changes.

• Base model license: Apache 2.0 — see the Qwen3-Embedding model card for the original license text.
• This quantization: Apache 2.0 (same terms as the base model).
• Modifications by Quadux IT GmbH: weight-only post-training quantization to the W8A16 (8-bit weights, 16-bit activations) scheme. No retraining, no fine-tuning, no architectural changes.

If you redistribute this model, you must include the Apache 2.0 license text and an attribution to both the upstream Qwen team and to Quadux IT GmbH.

Citation

The original Qwen3 Embedding work — please cite this if you publish results using this model:

@article{qwen3embedding,
  title  = {Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author = {Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal = {arXiv preprint arXiv:2506.05176},
  year   = {2025}
}

If you want to cite the quantization specifically:

@misc{{quadux_Qwen3-Embedding-4B-W8A16},
  author       = {{Quadux IT GmbH}},
  title        = {Qwen3-Embedding-4B-W8A16 (Quadux)},
  year         = {{2026}},
  publisher    = {{Hugging Face}},
  howpublished = {{\\url{{https://huggingface.co/QuaduxIT/Qwen3-Embedding-4B-W8A16}}}},
  note         = {{Post-training W8A16 (8-bit weights, 16-bit activations) quantization of Qwen/Qwen3-Embedding-4B}}
}

---

About Quadux IT GmbH

Software for engineering offices and accounting pipelines. Custom RAG infrastructure for internal Quadux deployments — these quants are the text-embedding side of that stack, released to the community as infrastructure we'd otherwise pay vendors for.

Find more at quadux.it · contact info@quadux.it

Imprint

Quadux IT GmbH · Schulstr. 3 · 37139 Adelebsen · Germany Registered Göttingen, HRB 206773 · VAT ID DE353975332 · DUNS 344198559 Managing Director: Walter Hoffmann