Instructions to use Qwen/Qwen3.8-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B

SGLang

How to use Qwen/Qwen3.8-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B
```

DFlash draft model instead of MTP one

#54

by artden111 - opened 30 days ago

Discussion

artden111

30 days ago

Is it possible? Will DFlash draft model be faster than MTP? Muse-Glimmer-30B uses DFlash draft model and it very fast

Manuun1

30 days ago

https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark

according to the sglang docs (https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B#2-configuration-tips). hope it helps 😃

artden111

29 days ago

Got this errors on llama.cpp

wrong number of tensors; expected 62, got 58

LinkuStarto

29 days ago

We really need a functional DFlash implementation. For some reason 30B Muse Glimmer runs almost 6-7 times faster than the new Qwen with DFlash . 5-6tok/s vs 50-60tok/s ... For a bit less intelligent model trade of I guess I will stick to the Muse Glimmer for now . Qwen 3.8 27B is really exceptionally good but not really usable with that speed on a 16gb vram

alex4o

29 days ago

btw guys just an instant idea I had (different dspar model per thinking level might be a smart thing that can be done)

kerrmetric

29 days ago

•

edited 29 days ago

btw guys just an instant idea I had (different dspar model per thinking level might be a smart thing that can be done)

I'd really love to see somebody test a different dspark or mtp model for the different thinking levels. It's got to have a small but meaningful impact right.

Avifenesh

29 days ago

i went the other way, model's own mtp head, trimmed on its own gens. 140 tok/s spec on one 6000.
draft: https://huggingface.co/Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF
engine: https://github.com/avifenesh/memra
https://inference.tiyuvta.ai/app

akierum

28 days ago

DFlash is not better than MTP, the MTP is 2x or 3x faster on token ingestion the Dflash is trash there, using it defeats the point of faster generation at all.

kerrmetric

28 days ago

DFlash is not better than MTP, the MTP is 2x or 3x faster on token ingestion the Dflash is trash there, using it defeats the point of faster generation at all.

DFlash in theory should be faster with enough compute (vs. bandwidth), and code-like usecases (where it's easy to specuate reasonably right tokens). Of course, in many real usecases / hardware you won't see that lift

mrchuy

28 days ago

•

edited 28 days ago

Managed to get one working with 50% first token acceptance based on 3.6 since they share the same shape.. Its slower than the stock MTP, but faster than plain https://huggingface.co/mrchuy/Qwen3.8-27B-DFlash-drafter-bootstrap-GGUF

Mode	Draft depth	Prompt tok/s	Decode tok/s	vs Plain	Draft acceptance	llama.cpp `mean len`	Acceptance by position
Plain Qwen3.8 Q4	—	1008.97	23.73	baseline	—	—	—
Qwen3.6 → Qwen3.8 DFlash	3	783.93	31.94	+34.6%	34.04%	2.02	57.2%, 29.7%, 15.2%
Native Qwen3.8 MTP	3	959.32	50.70	+113.7%	71.65%	3.15	85.5%, 71.0%, 58.5%

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment