Itopoly/G9v3-39A5B-W4A16

G9v3-39A5B — W4A16 GPTQ quant for vLLM

A serving-ready W4A16 GPTQ quantization of ai9stars/G9v3-39A5B (39B MoE, ~5.4B active params/token, 131K context, think/no-think modes, tool calling), produced with llm-compressor. To our knowledge this is the only vLLM/transformers-compatible quant of this model — the public alternatives (GGUF, MLX) cannot be served by vLLM for this architecture. Verified in production on 2× NVIDIA RTX 4000 Ada (20 GB each, TP=2).

Format	Size	Fits
BF16 (original)	73 GiB	80 GB+ GPU
GGUF / MLX (public quants)	—	llama.cpp / Mac only — not loadable by vLLM for this arch
this quant (W4A16)	~21 GB	2× 20 GB (TP=2) ✅ measured · 1× 24 GB (≤40K ctx) · 1× 40/48 GB · A100/H100

What's in this repo

File	What it is
model-0000{1,2}-of-00002.safetensors + model.safetensors.index.json	the quantized weights (~21 GB)
config.json	model config incl. the tp_plan fix (see below)
modeling_g9v3.py, configuration_g9v3.py	remote code incl. both required patches
tokenizer.json, tokenizer_config.json, generation_config.json	tokenizer
chat_template.jinja	default chat template
g9v3_chat_template_low.jinja / g9v3_chat_template_medium.jinja	templates with reasoning_effort default low/medium + legacy tool-history normalization
recipe.yaml	the GPTQ recipe used to produce this quant
G9V3-SETUP.md, G9V3-GIBBERISH-FIX.md	full setup guide + root-cause write-up of the GQA sharding bug

The two patches that MUST be in the modeling code (already applied)

If you ever replace modeling_g9v3.py from an upstream copy, re-apply or serving breaks:

1. G9v3MoE.moe() — delegate to vLLM's fused MoE runner: if not isinstance(self.experts, nn.ModuleList): return self.experts(hidden_states, topk_indices, topk_weights) Without it serving crashes (TypeError: object of type 'TransformersMoERunner' has no len()).
2. G9v3Model.__init__ — expose the tp_plan to vLLM's backend: self.tp_plan = getattr(config, "tp_plan", None) or {} Without it, output is gibberish — the transformers backend mis-shards GQA attention (K/V replicated instead of colwise). Full analysis in G9V3-GIBBERISH-FIX.md.

Also clear stale code caches when the modeling file changes: rm -rf ~/.cache/huggingface/modules/transformers_modules/*

How it was quantized

• llm-compressor GPTQ, W4A16, pipeline="sequential", sequential_targets=["G9v3DecoderLayer"]
• Calibration: ultrachat-200k, 512 samples, seq len 2048, dampening_frac=0.1
• moe_calibrate_all_experts=True — forces all 320 routed experts to see calibration tokens; without it, rarely-routed experts get no GPTQ statistics and degrade to round-to-nearest (the known weakness of public AWQ-style quants)
• Runtime: 2 h for 39 stages on 2× RTX 4000 Ada (4 samples/s/layer)
• The BF16 source does not fit in RAM on smaller boxes: load with device_map="cpu" plus an 80 GB swapfile (not device_map="auto" — compressed-tensors' offload converter asserts on transformers≥5 disk-offload tensors)

Serving (verified config, vLLM 0.28)

vllm serve G9v3-39A5B-W4A16 \
  --model-impl transformers --trust-remote-code \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 --max-num-seqs 8 \
  --enforce-eager \
  --chat-template g9v3_chat_template_low.jinja \
  --enable-auto-tool-choice --tool-call-parser minicpm5 \
  --reasoning-parser qwen3 --port 8000

Key facts:

• --model-impl transformers is required — g9v3 is not in vLLM's native registry; it serves through the transformers fallback backend (--trust-remote-code).
• --tensor-parallel-size is capped at 2 for this architecture (colwise split needs both the 32 query heads and 2 KV heads divisible by the TP degree). TP ∈ {1, 2} only.
• Tool calling: g9v3 emits <tool_call>… with <function name="..."> — the registered parser name in vLLM 0.28 is minicpm5.
• Reasoning field: vLLM 0.28 returns message.reasoning / streams delta.reasoning — NOT reasoning_content. Clients expecting the old field see no reasoning.
• The bundled chat templates additionally normalize legacy OpenAI function_call/role: "function" history into modern tool_calls/role: "tool" — without this, clients sending the legacy format drop tool history and the model loops on the same call forever.

Verify after install

curl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"g9v3-39a5b","messages":[{"role":"user","content":"What is the capital of France? Answer briefly."}],"max_tokens":100,"temperature":0}' \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

Expected: a coherent short answer ("The capital of France is Paris."). Gibberish means the tp_plan patch didn't land; a TransformersMoERunner TypeError means patch (1) is missing.

Measured deployment (2× RTX 4000 Ada, 20 GB each)

• Weights: 8.95 GiB/GPU · KV cache: 7.03 GiB/GPU (gpu-mem-util 0.90, 32K ctx)
• Concurrency: ~6× full-length 32K streams, capped at 8 sequences
• Same tp_plan + patches serve every GPU tier — only the model dir differs (see the single-card tiers in the table above)

Credits

• Original model: ai9stars/G9v3-39A5B (Apache-2.0)
• Quantization, patches, and validation: Itopoly
• CPU/Mac alternatives: mradermacher GGUFs · MLX 4-bit