randmaru/Spark-X2.6-4B-mlx-mxfp4

An MXFP4 MLX quantization of XHToken/Spark-X2.6-4B for Apple Silicon inference, produced with mlx-lm 0.32.0 / mlx 0.32.3.

Weights are quantized to 4-bit float (MXFP4, group size 32, shared E8M0 exponent per group); the per-head attention output gate (self_attn.g_proj) is kept in BF16. The result is a single model.safetensors of 2.19 GB (2.04 GiB), averaging ≈4.25 bits per weight — 3.76× smaller than the BF16 checkpoint. See the MXFP8 sibling build if you need higher fidelity: it is ~2× larger but nearly lossless.

Quality heads-up. On the fixed-text evaluation described below, MXFP4 costs +24% perplexity on this model, noticeably more than plain affine 4-bit (+4.5%). It stays coherent and is fine for drafting, tool routing and other forgiving workloads, but it is the lossy end of this family — prefer MXFP8 or full precision when output quality matters. Details and per-block attribution are in Quality.

This is an unofficial, community quantization and is not affiliated with or endorsed by XHToken. All model weights, architecture, and license terms come from the base model.

Model details

From the base model card:

• Parameters: 4.11B, dense (no MoE)
• Architecture: 36 decoder layers — 27 sliding-window layers (window 512) and 9 full-attention layers in a 3:1 pattern; hidden_size 2560, intermediate_size 10240, GELU MLP
• Attention: GQA with 16 query heads and 4 KV heads, head_dim 256; a per-head sigmoid output gate (g_proj)
• RoPE: full layers use rope_theta 5,000,000 with partial_rotary_factor 0.25; sliding layers use 10,000 with full rotary
• Vocabulary: 131,072, embeddings tied to the output head
• Context length: 1,048,576 tokens
• Thinking budget: three levels (none / low / high), controlled through the chat template — enable_thinking and thinking_effort
• Languages: English, Chinese
• Type: text-only on-device agent model (no vision tower)

Quantization

Property	Value
Mode	mxfp4 (4-bit float, shared E8M0 exponent per group)
Group size	32
Weight storage	U32 packed E2M1 weights + U8 E8M0 scales
Scale overhead	1 byte per 32 weights → 4.25 bpw
Excluded from quantization	self_attn.g_proj (per-head gate) → BF16
Embeddings	quantized (tied to the output head)
Unquantized	all norms (1-D) in BF16
Effective bits per weight	≈4.25
Output layout	a single model.safetensors (no shards)

The g_proj exception is the model's own quant_predicate inside mlx-lm, not a manual tweak: the gate is a tiny 16×2560 projection whose sigmoid values are sensitive to rounding.

Where the bytes go

Component	Bytes	Share
Packed 4-bit weights (U32)	2,055,208,960 (2.06 GB)	94.0%
E8M0 scales (U8, one per 32 weights)	128,450,560 (128.5 MB)	5.9%
BF16 unquantized (g_proj, norms)	3,322,880 (3.3 MB)	0.2%

Size comparison

There is no official MLX build of this model, so the closest published references are the official Transformers-format quants from XHToken. They target a different runtime (GPU, transformers/vLLM/SGLang), so the comparison is indicative only — sizes are not directly comparable on Apple Silicon.

Build	Format / runtime	Weights	Notes
mlx-mxfp4 (this repo)	MXFP4, mlx-lm	2.19 GB	g_proj + norms in BF16
mlx-mxfp8	MXFP8, mlx-lm	4.24 GB	same pipeline, near-lossless
XHToken/Spark-X2.6-4B-INT4	affine INT4, Transformers	3.41 GB	official
XHToken/Spark-X2.6-4B-FP8	FP8, Transformers	4.45 GB	official
XHToken/Spark-X2.6-4B-INT8	affine INT8, Transformers	4.45 GB	official
XHToken/Spark-X2.6-4B	BF16, Transformers	8.22 GB	reference
XHToken/Spark-X2.6-4B-GGUF	Q4_K_M, llama.cpp	3.54 GB	different runtime

MXFP4 is the smallest option here because it needs only one byte of scale per 32 weights and no per-group bias term. The official INT4 build is ~1.2 GB larger: integer quantization with 32-weight groups carries a BF16 scale and a BF16 bias per group (≈5 bpw), while MXFP4 carries an 8-bit exponent only.

Quality

Measured on a fixed corpus of 12 English and 3 Chinese Wikipedia articles (plain-text extracts, 203,577 tokens): non-overlapping 1024-token windows, perplexity over all predicted tokens, plus top-1 agreement — the share of positions where the quantized build's argmax token matches the BF16 build.

Build	Perplexity	Δ vs BF16	Top-1 agreement
BF16 (reference)	13.513	—	100%
MXFP8	13.528	+0.11%	93.2%
affine 4-bit, group 64 (diagnostic, not published)	14.127	+4.5%	80.8%
MXFP4 (this repo)	16.809	+24.4%	75.0%

These are not benchmark scores and are not comparable to published WikiText numbers; they only quantify how far each build drifts from the BF16 weights of the same model on the same text.

Where the MXFP4 loss comes from

Keeping individual blocks in BF16 and quantizing the rest shows that the penalty is concentrated in the MLP projections:

MXFP4 variant	Perplexity	Δ vs BF16	Top-1
full MXFP4 (this repo)	16.809	+24.4%	75.0%
MLP projections kept BF16	14.375	+6.4%	81.3%
attention projections kept BF16	15.176	+12.3%	80.1%
embedding / head kept BF16	16.926	+25.3%	76.3%

The embedding (which is also the output head) is not the culprit; MXFP4's exponent-sharing on the MLP weights is. A mixed build with BF16 MLP would land close to affine 4-bit quality, but at ~6.3 GB (≈12 bpw) — most of the size advantage gone — which is why only the plain MXFP4 and MXFP8 builds are published.

Speed

Apple M5 Pro (24 GB), mlx 0.32.3 / mlx-lm 0.32.0. 322-token prefill, then 128 greedy decode steps; prefill is materialized before decode timing starts; median of 3 runs.

Build	Prefill tok/s	Decode tok/s	Peak memory
BF16	2,169	31.9	8.60 GB
MXFP4	2,023	96.5	2.83 GB
MXFP8	1,813	55.6	4.76 GB

Requirements

mlx-lm 0.32.0 or newer — that is the first release with the spark2_5 architecture (mlx_lm/models/spark2_5.py; 0.31.3 fails with Model type spark2_5 not supported):

pip install -U "mlx-lm>=0.32.0"

mlx-lm 0.32.0 requires mlx>=0.32.2, which provides the mxfp4 quantization mode used here. Built and tested with mlx 0.32.3.

Usage

mlx_lm.generate --model ./Spark-X2.6-4B-mlx-mxfp4 \
  --prompt "What is the capital of Anhui Province?" --max-tokens 512

OpenAI-compatible server:

mlx_lm.server --model ./Spark-X2.6-4B-mlx-mxfp4

Python, with the base model's chat template:

from mlx_lm import load, generate

model, tokenizer = load("./Spark-X2.6-4B-mlx-mxfp4")

messages = [{"role": "user", "content": "What is the capital of Anhui Province?"}]
prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,       # False for direct answers
    thinking_effort="low",      # "low" | "high"
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=1024))

Recommended sampling from the base model: temperature 1.0, top_p 0.95. Default context is 1M tokens, so cap max_tokens (or the server's context) to what you actually need.

Files

File	Size
model.safetensors	2,187,035,935
tokenizer.json	10,115,785
model.safetensors.index.json	33,744
LICENSE	11,338
config.json	2,503
chat_template.jinja	5,717
tokenizer_config.json	420
generation_config.json	281

configuration_spark.py and modeling_spark.py are copied from the source checkpoint by mlx_lm.convert; they are only used by the transformers custom-code path and are not needed for MLX inference.

License

Apache-2.0, inherited from XHToken/Spark-X2.6-4B — see the LICENSE file.

Citation

@misc{spark2026x26,
  title  = {Spark-X2.6-4B: Advancing the Frontier of On-Device Agents for Real-World Tasks},
  author = {XHToken},
  year   = {2026},
  url    = {https://huggingface.co/XHToken/Spark-X2.6-4B}
}