Instructions to use Qwen/Qwen3.8-Flash-Next with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-Flash-Next with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-Flash-Next")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-Flash-Next")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-Flash-Next", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-Flash-Next with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-Flash-Next"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-Flash-Next",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-Flash-Next

SGLang

How to use Qwen/Qwen3.8-Flash-Next with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-Flash-Next" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-Flash-Next",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-Flash-Next" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-Flash-Next",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-Flash-Next with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-Flash-Next
```

Just the best model 💘 as always Qwen Team is number 1 🥰 .

#51

by auf1r2 - opened Sep 7

Discussion

auf1r2

Sep 7

•

edited Sep 7

Just facts:

I can run 170Gb Q6_K_XL on a single Strix Halo 128Gb mini PC 20t/s TG, 300+ t/s PP (no MTP) ~ thanks to new LLM architecture
It beats every other local model of that size in coding. Very clever model, you can delegate it some work and it won't stuck like other models can do.
Default xhigh settings for reasoning is clever decision, it may start slowly but if you calculate total time to delivering fully functional code - it is faster! Other "fast" models are fast to deliver garbage but if you try to deliver fully functional code - Qwen3.8 FN on XHigh settings is faster! (I'm talking about real project production code, not funny 1 HTML test apps, but it's good for them too)

Love Qwen models, cheers to Qwen Team for contributing local AI community 🥂

bidyut12345

Sep 8

•

edited Sep 8

Hi, can you share how you managed to run Q6_K_M. I am having hard time to go beyond Q3_k_XL from unsloth. I am on windows.

auf1r2

Sep 8

•

edited Sep 8

I'm using Ubuntu 26.04 on Strix Halo

Set 64GB VRAM + 64GB RAM in BIOS

Ubuntu params for Grub (/etc/default/grub): GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=off amdgpu.gttsize=61440 amdgpu.ppfeaturemask=0xffffffff pcie_aspm=off nvme_core.default_ps_max_latency_us=0 pcie_port_pm=off amdgpu.gpu_recovery=1 amdgpu.lockup_timeout=60000 amdgpu.cik_support=0 amdgpu.si_support=0 rocm_debug=0" (after editing you need to call sudo update-grub -> restart) <<--- This tells OS to use some Strix Halo optimizations + allows it to use 64 + 60 = 124 GB of memory for VRAM.

Install amd-ttm -> run amd-ttm --set 60 then restart.

then command to run llama-server:
./llama-server --model "/home/xxx/EvoX2/LLMs/models/unsloth/Qwen3.8-Flash-Next-GGUF-Q6/Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf" --host 192.168.1.xx --port 8080 --tools get_info --cors-origins 192.168.1.xx --api-key xxx --jinja --n-gpu-layers all --n-cpu-moe 0 --parallel 1 --no-kv-unified --kv-offload --load-mode mmap --spec-type none --fit off --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --ctx-size 163840 --batch-size 2048 --ubatch-size 2048 --no-op-offload --override-tensor "per_layer_token_embd=CPU" --lazy-mode on --temperature 1.0 --top-k 20 --repeat-penalty 1.00 --presence-penalty 0.0 --top-p 0.95 --min-p 0.0 --reasoning on --chat-template-kwargs '{"reasoning_effort":"xhigh"}' --reasoning-preserve

bidyut12345

Sep 9

Clever trick, thanks for the tips! It helped me step up to Q4_K_XL. I'm on Windows and use this machine for work, so switching to Ubuntu or allocating the entire 124 GB of RAM isn't an option. Still, with your optimizations, getting Q4_K_XL running is a great result. Appreciate it!

joonanykanen

Sep 9

Just facts:

I can run 170Gb Q6_K_XL on a single Strix Halo 128Gb mini PC 20t/s TG, 300+ t/s PP (no MTP) ~ thanks to new LLM architecture

It beats every other local model of that size in coding. Very clever model, you can delegate it some work and it won't stuck like other models can do.

Default xhigh settings for reasoning is clever decision, it may start slowly but if you calculate total time to delivering fully functional code - it is faster! Other "fast" models are fast to deliver garbage but if you try to deliver fully functional code - Qwen3.8 FN on XHigh settings is faster! (I'm talking about real project production code, not funny 1 HTML test apps, but it's good for them too)

Love Qwen models, cheers to Qwen Team for contributing local AI community 🥂

Can endorse this. Running the Intel/Qwen3.8-Flash-Next-W4A16-AutoRound quant and been absolutely stunned how good the model is in doing what you tell it to do. The throughput is amazing as well due to the small number of activate params and being able to run the important stuff only on VRAM (with the n-gram-table offload). Sometimes it feels like it can almost read your mind. Feeling very Opus-like, in some tasks even better. I really cannot wait for the Qwen4-family releases – in hopes that they would release an even better trained checkpoint than this with the same architecture!!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment