Instructions to use VnimanieAI/Qwen3.8-Flash-Next-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use VnimanieAI/Qwen3.8-Flash-Next-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="VnimanieAI/Qwen3.8-Flash-Next-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("VnimanieAI/Qwen3.8-Flash-Next-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("VnimanieAI/Qwen3.8-Flash-Next-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use VnimanieAI/Qwen3.8-Flash-Next-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "VnimanieAI/Qwen3.8-Flash-Next-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VnimanieAI/Qwen3.8-Flash-Next-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/VnimanieAI/Qwen3.8-Flash-Next-W4A16

SGLang

How to use VnimanieAI/Qwen3.8-Flash-Next-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "VnimanieAI/Qwen3.8-Flash-Next-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VnimanieAI/Qwen3.8-Flash-Next-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "VnimanieAI/Qwen3.8-Flash-Next-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VnimanieAI/Qwen3.8-Flash-Next-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use VnimanieAI/Qwen3.8-Flash-Next-W4A16 with Docker Model Runner:
```
docker model run hf.co/VnimanieAI/Qwen3.8-Flash-Next-W4A16
```

Get endless model loading with 8x3090

by leflak - opened Aug 28

Discussion

leflak

Aug 28

Hello, thank you for the model quant. I use the exact same command as in the repo but near to the end of model loading process I get endless waiting juste after:
(Worker_TP2_EP2 pid=1170) INFO 08-28 11:17:47 [gpu_worker.py:919] Free memory on device (23.1/23.56 GiB) on startup. Desired GPU memory utilization is (0.9, 21.2 GiB). Actual usage is 12.14 GiB for consumed memory (weights + non-torch), 0.9 GiB for peak activation, and 0.98 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with --kv-cache-memory=7555032167 (7.04 GiB) to fit into requested memory, or --kv-cache-memory=9596911104 (8.94 GiB) to fully utilize gpu memory. Current kv cache memory in use is 8.16 GiB.
(EngineCore pid=892) INFO 08-28 11:18:13 [shm_broadcast.py:801] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).

Any idea?

Command used:
docker run --name flashnext --gpus all --shm-size 32g
--cap-add SYS_PTRACE --security-opt seccomp=unconfined
-v /home/huh/Modèles/Qwen3.8-Flash-Next-W4A16:/model -p 8000:8000
-e VLLM_PLE_CPU_OFFLOAD=1
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
vllm/vllm-openai:qwen38-flash-next
--model /model --served-model-name Qwen3.8-Flash-Next
--tensor-parallel-size 8 --enable-expert-parallel
--gpu-memory-utilization 0.90 --max-model-len 262144 --max-num-seqs 64
--compilation-config '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}'
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'
--reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice
--trust-remote-code

AB-VnimanieAI

Vnimanie LLC org Aug 30

I have similar sounding troubles a year before when I was using 4 x Gigabyte 3090 Turbo + 4 x Palit 3090. May that be that your cards somehow have less memory? Palit cards usually do. If so, try to shorten length or change GPU memory utilization. This quant was made for Gigabyte 3090 Turbo cards.
I will be able to elaborate the day after tomorrow when my Fable limits will restore)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment