Instructions to use wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16

SGLang

How to use wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 with Docker Model Runner:
```
docker model run hf.co/wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16
```

ThinkingCap-Qwen3.8-27B — AutoRound INT4 (W4A16)

An INT4 AutoRound quantization of bottlecapai/ThinkingCap-Qwen3.8-27B, BottleCap AI's reasoning fine-tune of Qwen/Qwen3.8-27B. It is 19.0 GB and keeps the MTP head for self-speculative decoding.

BottleCap publishes FP8, NVFP4 and GGUF builds, but no INT4 AutoRound. This one exists so the model can run on two 24 GB cards with the same tiers as the base Qwen3.8-27B in club-3090.

Quantization


Scheme	W4A16: int4, symmetric, group size 128, `auto_round:auto_gptq` packing
Kept in BF16	`lm_head`, embeddings, vision tower, GDN `linear_attn.in_proj_a` / `in_proj_b`, `mtp.fc`, all norms
MTP head	`mtp.fc` in BF16; the MTP layer's 7 linear weights in int4 (round-to-nearest, not tuned)
Tool	auto-round 0.14.2
Tuning	200 iterations, 512 samples, seqlen 2048, `enable_alg_ext=True`
Calibration	auto-round's default (`NeelNanda/pile-10k`)

The recipe is also stored in club3090_build_recipe.json.

The packed layout is tensor-for-tensor identical to Frozenlock/Qwen3.8-27B-int4-AutoRound: the same quantized and BF16 module families, and the same MTP tensors. So it should load anywhere that checkpoint does. Only the tuning differs: 512 samples with alg_ext here, where that card says "default tuning".

Serving (vLLM)

vllm serve wasifb/ThinkingCap-Qwen3.8-27B-AutoRound-W4A16 \
  --quantization auto_round --dtype bfloat16 --tensor-parallel-size 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Sampling follows the base model: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0. Thinking effort is set with chat_template_kwargs: {"reasoning_effort": "xhigh"}, which is the template's default.

Known upstream issues on stock vLLM, each fixed in club-3090's composes:

MTP with prefix caching on Qwen3-Next hybrids can corrupt the recurrent state (vllm#43559; fix vllm#48375 is still open). club-3090 vendors the fix. On stock vLLM, disable prefix caching when MTP is on.
W4A8 (VLLM_MARLIN_INPUT_DTYPE=int8, roughly +40–50% prefill in our Ampere measurements) doesn't work on stock vLLM for AutoRound checkpoints. The setting is silently ignored (vllm#48904), and negative group scales corrupt the output (vllm#48905). Both are patched in club-3090. On stock vLLM, serve it as W4A16.

Validation

Tested on 2× RTX 3090 (PCIe, 230 W cap) with vLLM v0.30.0 through club-3090's dual-fast configuration: TP=2, W4A8, MTP n=4, fp8 KV, 262K context. Compared against Frozenlock's INT4 of the base Qwen3.8-27B on the same rig and image, the same day.

	this quant
verify-full	pass
Decode, narrative / code (tok/s)	74.0 / 101.0
Prefill @10K / @90K (tok/s)	1,720 / 1,246
MTP acceptance length, n=4	3.6–5.0 (bench); 2.7–4.0 throughout the 20K run
20,000-token forced generation	no drafter collapse, 0 restarts

These are decode and serving checks. The quality evaluation, club-3090's 8-pack, is pending, and this card will be updated with it.

License

ThinkingCap is licensed under the PolyForm Small Business License 1.0.0, plus BottleCap AI's personal-use permission; see LICENSE. This quantization is a derivative and is distributed under the same terms. Commercial use by organizations that don't qualify as a small business needs a commercial license from BottleCap AI.

ThinkingCap is derived from Qwen materials, which are licensed under Apache-2.0. See NOTICE and LICENSE-Apache-2.0-Qwen.txt.

All credit for the model goes to BottleCap AI and the Qwen team. This repository only adds the quantization.