Support this work β†’ Β· X Β· GitHub Β· REAP paper Β· Cerebras REAP

Qwen3.5-88B

REAP-pruned Qwen/Qwen3.5-122B-A10B.

At a glance

Base model Qwen/Qwen3.5-122B-A10B
Format BF16
Total params 88B
Active / token 10B
Experts / layer β€”
Layers β€”
Hidden size β€”
Context β€”
On-disk size 175 GB

Which variant should I pick?

Variant Format Link
Qwen3.5-264B BF16 link
Qwen3.5-264B-FP8 FP8 link
Qwen3.5-264B-W4A16 W4A16 link
Qwen3.5-28B BF16 link
Qwen3.5-35B-EXL3-4bpw EXL3-4bpw link
Qwen3.5-76B BF16 link
Qwen3.5-76B-GGUF GGUF link
Qwen3.5-88B (this) BF16 link
Qwen3.5-99B BF16 link
Qwen3.5-99B-GGUF GGUF link

30% expert-pruned variant of Qwen3.5-122B-A10B using REAP (Routing-Enhanced Activation Pruning).

Model Details

Property Value
Base Model Qwen/Qwen3.5-122B-A10B
Architecture Qwen3.5 MoE (GDN + Full Attention)
Original Experts 256 per layer
Pruned Experts 180 per layer (30% removed)
Active Parameters ~10B per token
Pruning Method REAP with targeted refusal preservation
Preserve Threshold 80% (super-expert protection)
Calibration reap-calibration-data-v1 β€” 23k benchmark-free samples
Maintainer 0xSero
Organization Sybil Solutions
Project REAP PR17

Usage

vllm serve 0xSero/Qwen3.5-88B \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-model-len 8192 \
  --trust-remote-code \
  --language-model-only \
  --dtype bfloat16

Important: Use --language-model-only flag β€” this is a text-only checkpoint pruned from the multimodal base model.

What is REAP?

REAP (Routing-Enhanced Activation Pruning) removes the least-activated experts from MoE models while preserving critical capabilities. It uses router activation patterns from a calibration dataset to identify dispensable experts, with special protection for safety-critical behaviors.

License

Same license as the base model (Qwen).

License & citation

License inherited from the base model.

@misc{lasby2025reap,
  title  = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression},
  author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa},
  year   = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv}
}

Sponsors

Made possible by NVIDIA Β· TNG Technology Β· Lambda Β· Prime Intellect Β· Hot Aisle.

Downloads last month
28
Safetensors
Model size
88B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for 0xSero/Qwen3.5-88B

Finetuned
(59)
this model
Quantizations
3 models

Collection including 0xSero/Qwen3.5-88B

Paper for 0xSero/Qwen3.5-88B