Qwen-3.5 Collection
Collection
Quantized Qwen3.5 models for efficient image-text understanding (AutoRound W4A16). • 6 items • Updated
• 1
Vishva007/Qwen3.5-9B-W4A16-AutoRound-AWQ
This is a W4A16 (4-bit weight, 16-bit activation) AWQ-format quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention.
Quantization Details
| Parameter | Value |
|---|---|
| Method | AutoRound (W4A16, AWQ format) |
| Group Size | 128 |
| Symmetric | Yes |
| Iterations | 800 |
| Calibration Samples | 512 |
| Sequence Length | 2048 |
| Torch Compile | Enabled |
Key Notes
- AWQ format — Exported in the standard AWQ format, optimized for efficient inference with activation-aware weight clipping.
- High accuracy configuration — 800 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
- W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
- ~50% memory reduction compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
Usage
This model is compatible with transformers, AutoAWQ, vLLM, and SGLang — any backend supporting AWQ-format weights works out of the box. For full model details, architecture, and capabilities, refer to the base model page.
- Downloads last month
- 78
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support