Instructions to use tuevuong/Spark-X2.6-4B-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tuevuong/Spark-X2.6-4B-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tuevuong/Spark-X2.6-4B-NVFP4A16", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("tuevuong/Spark-X2.6-4B-NVFP4A16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tuevuong/Spark-X2.6-4B-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tuevuong/Spark-X2.6-4B-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tuevuong/Spark-X2.6-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tuevuong/Spark-X2.6-4B-NVFP4A16

SGLang

How to use tuevuong/Spark-X2.6-4B-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tuevuong/Spark-X2.6-4B-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tuevuong/Spark-X2.6-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tuevuong/Spark-X2.6-4B-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tuevuong/Spark-X2.6-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tuevuong/Spark-X2.6-4B-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/tuevuong/Spark-X2.6-4B-NVFP4A16
```

Spark-X2.6-4B: Advancing the Frontier of On-Device Agents for Real-World Tasks

This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format.

Spark-X2.6 benchmark comparison

Introduction

A little over a month has passed since Spark-X2.5. In the rapidly evolving world of AI agents, that is enough time to rethink, refine, and move the frontier forward. Building on the Spark-X2.5-4B, we introduce Spark-X2.6-4B, the latest generation of our compact, general-purpose language models for on-device deployment.

Spark-X2.6-4B retains exactly the same architecture as Spark-X2.5-4B, achieving its capability gains through improved post-training. It delivers substantial improvements in coding, agent, long-context, multilingual, reasoning, etc. These advances enable more capable and practical on-device AI for developers and users tackling real-world tasks.

Technical Highlights:

Stronger Coding, Agentic, and Long-Context Capabilities: The models deliver substantial improvements in agentic workflows involving complex tool use, multi-step planning, and software engineering. They achieve state-of-the-art performance among similarly sized open-source models, matching or even surpassing models with approximately 9B parameters.
Controllable Thinking Budget: The models provide three configurable thinking levels—none, low, high—allowing applications to balance reasoning quality, latency, and inference cost dynamically.

Benchmarks

We evaluate our models and compare them with leading on-device models of similar size across a broad range of tasks, including agent, code, math, general and knowledge. Spark-X2.6-4B achieves best-in-class performance among models of comparable size, while matching or exceeding the performance of larger 9B models on several benchmarks.

all Spark-X2.6 benchmarks

Edge Inference

Spark-X2.6-4B delivers inference efficiency on par with or better than comparable on-device models, including Qwen3.5-4B and Gemma-E4B. It combines this efficiency with stronger capabilities, making it a practical choice for responsive, resource-conscious on-device AI.

Spark-X2.6 inference

Quickstart — Docker on RTX 5070 Ti (Blackwell SM120, vLLM)

This repo is the NVFP4A16 weight-only quantization of XHToken/Spark-X2.6-4B (produced with llmcompressor model_free_ptq(scheme="NVFP4A16"), no calibration data). Verified on a 16 GB RTX 5070 Ti: 256k context within a ~10 GB VRAM budget.

1. Build the serving image

Spark2_5ForCausalLM is not in vLLM's core yet — the Spark plugin registers it as an out-of-tree general plugin. The three patches below fix API drift between the plugin (vendored from an older vLLM) and vLLM ≥ 0.32.

FROM vllm/vllm-openai:latest   # v0.32+, Blackwell-ready

RUN pip install --no-deps https://github.com/XHToken/Spark-plugin/archive/refs/heads/main.tar.gz \
 # patch 1: AutoWeightsLoader.__init__ lost skip_prefixes (always None here — tie_word_embeddings=True)
 && sed -i "/skip_prefixes=(/d" /usr/local/lib/python3.12/dist-packages/vllm_spark2_5_plugin/spark2_5.py \
 # patch 2: make_tool_call_id moved to vllm.renderers.chat_utils
 && sed -i "s#from vllm.entrypoints.chat_utils import#from vllm.renderers.chat_utils import#" /usr/local/lib/python3.12/dist-packages/vllm_spark2_5_plugin/spark2_5_tool_parser.py \
 # patch 3: re-export shim for the moved protocol symbols
 && mkdir -p /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/engine \
 && touch /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/engine/__init__.py \
 && printf "from vllm.entrypoints.generate.base.protocol import DeltaFunctionCall, DeltaToolCall, ExtractedToolCallInformation\nfrom vllm.entrypoints.openai.chat_completion.protocol import DeltaMessage, FunctionCall, ToolCall\n" > /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/engine/protocol.py \
 && python3 -c "import vllm_spark2_5_plugin.spark2_5, vllm_spark2_5_plugin.spark2_5_tool_parser; print('plugin OK')"

docker build -t vllm-openai:spark-x26 .

2. Serve

docker run -d --name spark-x26 \
  --gpus all --ipc=host \
  -p 8000:8000 \
  -v spark-x26-cache:/root/.cache \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  vllm-openai:spark-x26 \
  --model tuevuong/Spark-X2.6-4B-NVFP4A16 \
  --served-model-name spark-x2.6-4b \
  --trust-remote-code \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.60 \
  --kv-cache-dtype nvfp4 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser spark25

The first run downloads the checkpoint (~2.8 GB) into the mounted HF cache. Startup takes ~2–3 min (torch.compile + FlashInfer builds the fp4 KV kernels once — kept in the spark-x26-cache volume).

Why these exact flags on a 5070 Ti (16 GB):

Flag	Reason
`--kv-cache-dtype nvfp4`	The deal-maker for 256k in 10 GB. The model is hybrid attention (36 layers: 27 sliding-window `512` + 9 full). fp8 KV still needs 6.2 GiB for 262,144 tokens > budget; nvfp4 cuts it to ~3.1 GiB.
`--gpu-memory-utilization 0.60`	≈9.8 GB pool (~10.2 GB incl. CUDA context) — leaves the rest of the 16 GB card for a desktop / other inference server (e.g. ollama).
`--max-num-batched-tokens 4096`	With 16384 the FlashInfer fp4-prefill workspace (~640 MB) does not fit in the headroom left by the KV pool → engine OOMs on the first long request. 4096 is safe; prefill is chunked, so full 256k stays usable.
`--tool-call-parser spark25`	Spark's XML/KV tool-call format (`<tool_call>name<arg_key>…`), registered by the plugin.

Success line to look for: GPU KV cache size: 392,466 tokens … Maximum concurrency for 262,144 tokens per request: 1.50x.

3. Test

# chat (thinking off; on by default)
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "spark-x2.6-4b",
  "messages": [{"role": "user", "content": "What is the capital of France?"}],
  "max_tokens": 128,
  "chat_template_kwargs": {"enable_thinking": false}
}'

# tool calling
curl -s localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "spark-x2.6-4b",
  "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
  "tools": [{"type": "function", "function": {"name": "get_weather",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
  "max_tokens": 128
}'

Long context check (needle recall measured at 258,933 prompt tokens, exact-match answer):

python3 - <<'PY'
import json, urllib.request
filler = "The quick brown fox jumps over the lazy dog while the sun sets behind the quiet hills. "
segments = [str(i) + ": " + filler for i in range(10800)]
segments[6000] = "IMPORTANT FACT: the secret launch passcode for mission ORION-7 is ZEBRA-PINEAPPLE-4217. "
prompt = "".join(segments) + "Question: What is the secret launch passcode for mission ORION-7? Answer concisely."
body = {"model": "spark-x2.6-4b", "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 200, "temperature": 0.0,
        "chat_template_kwargs": {"enable_thinking": False}}
req = urllib.request.Request("http://localhost:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
r = json.load(urllib.request.urlopen(req, timeout=1500))
print(r["usage"]["prompt_tokens"], "tokens →", r["choices"][0]["message"]["content"])
PY

Troubleshooting (SM120 / 5070 Ti specifics)

Failed to initialize NVML: Unknown Error inside the container (Docker 29 + runc ≥1.5 + nvidia-container-toolkit 1.20.x on some hosts — the hook's cgroup-v2 device rules are dropped). Add explicit device passthrough to docker run:
```
--device /dev/nvidia0 --device /dev/nvidiactl --device /dev/nvidia-modeset \
--device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools
```
Architecture Spark2_5ForCausalLM not supported → the plugin layer didn't build (step 1). The build must print plugin OK.
Don't use XHToken/Spark-X2.6-4B-INT4 on this GPU — its W8A8-INT8 modules have no SM120 kernel (Int8 not supported on SM120, even with the Triton fallback the mixed-precision path leaves no room for a 256k KV cache). This NVFP4 checkpoint is the SM120-native alternative.
Out of VRAM with other GPU users? --gpu-memory-utilization is a fraction of the total 16 GB — lower it until the engine reports ≥262,144 KV tokens, or shorten --max-model-len.

License

The Spark-X2.6 model series is licensed under the Apache 2.0 License.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{sparkx2.6,
    title  = {Spark-X2.6-4B: Advancing the Frontier of On-Device Agents for Real-World Tasks},
    author = {SparkLLM Team},
    year   = {2026}
}

Downloads last month: 161

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for tuevuong/Spark-X2.6-4B-NVFP4A16

Base model

XHToken/Spark-X2.6-4B-Base

Quantized

(5)

this model