Instructions to use Ninnix96/Ling-3.0-tiny-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Ninnix96/Ling-3.0-tiny-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4
# Run inference directly in the terminal:
llama cli -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4
# Run inference directly in the terminal:
llama cli -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4
# Run inference directly in the terminal:
./llama-cli -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Use Docker

docker model run hf.co/Ninnix96/Ling-3.0-tiny-gguf:MXFP4

LM Studio
Jan

vLLM

How to use Ninnix96/Ling-3.0-tiny-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ninnix96/Ling-3.0-tiny-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ninnix96/Ling-3.0-tiny-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Ollama
How to use Ninnix96/Ling-3.0-tiny-gguf with Ollama:
```
ollama run hf.co/Ninnix96/Ling-3.0-tiny-gguf:MXFP4
```
Unsloth Desktop

How to use Ninnix96/Ling-3.0-tiny-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Ninnix96/Ling-3.0-tiny-gguf:MXFP4"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Ninnix96/Ling-3.0-tiny-gguf with Docker Model Runner:
```
docker model run hf.co/Ninnix96/Ling-3.0-tiny-gguf:MXFP4
```

Lemonade

How to use Ninnix96/Ling-3.0-tiny-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Run and chat with the model

lemonade run user.Ling-3.0-tiny-gguf-MXFP4

List all available models

lemonade list

Hermes Agent

How to use Ninnix96/Ling-3.0-tiny-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Ninnix96/Ling-3.0-tiny-gguf with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Ninnix96/Ling-3.0-tiny-gguf:MXFP4

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Ninnix96/Ling-3.0-tiny-gguf:MXFP4" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

antirez-recipe quants

MoE quants following the antirez recipe: routed experts quantized hard, everything else protected.

Tensor class	Quant	Notes
`blk..ffn_gate_exps`, `blk..ffn_up_exps` (`ffn_gate_up_exps` if fused)	Q5_K (imatrix)	routed-expert gate/up
`blk.*.ffn_down_exps`	Q8_0 (imatrix)	routed-expert down (K-quant for quality)
`blk..ffn_{gate,up,down}_shexp`, all `attn_`, `output.weight`	`Q8_0`	shared experts / attention / head
`token_embd.weight`	`F16`	embeddings
router (`ffn_gate_inp`), norms, sinks, biases	source precision (`BF16`/`F32`)	never quantized by llama.cpp

Base type Q8_0 + --tensor-type overrides, imatrix calibration (see SPEC per-model imatrix links). Runs on upstream llama.cpp — no fork or patch needed.

Validation per file: gguf-dump tensor audit (experts = GU/DOWN, protection intact), llama-cli smoke load, llama-perplexity spot ppl + --kl-divergence vs BF16 logits.

Files

File	Size	SHA-256
`Ling-3.0-tiny-Q4K-MXFP4-Q8.gguf`	5091030464	`54859c79a6700c944b686bade6ef3ec062f1ca06c31a32fdeac8d2b350c870b4`
`Ling-3.0-tiny-Q5_K_M.gguf`	6898091232	`34bd66526660e6d9c02b0f592cd2c0a95fc17b24b9345900d4d95061a8f18d07`

Imatrix: https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF/Ling-3.0-tiny-imatrix.gguf

Recipe

Antirez recipe on inclusionAI/Ling-3.0-tiny (BF16 source: inclusionAI/Ling-3.0-tiny-GGUF, Ling-3.0-tiny-bf16.gguf):

ffn_gate_exps / ffn_up_exps -> Q5_K
ffn_down_exps -> Q8_0
attention + shared experts + output -> Q8_0
token embeddings -> F16
imatrix: bloomer010/Ling-3.0-tiny-GGUF (Ling-3.0-tiny-imatrix.gguf, sha-verified identical locally)

15,065 MiB BF16 -> 6,572 MiB (6.98 BPW).

Tokenizer note: Ling-3.0-tiny ships its own chat template (tokenizer.ggml.pre = bailingmoe2) — it works as-is on upstream llama.cpp, no extra flags needed.

Validation (llama.cpp, Ling arch)

12-case hard-smoke eval (temp 0, ctx 15360, full-resident -ngl 999, AMD Vulkan/UMA): 4/12 PASS, ~120-124 t/s. Fails 2826, 10360, 6034, 10567, 9487, 1606, 1716, cwe121 (long-search loops that never commit a final answer; generations cut mid-thought). Passes 6500, 1618 and both big-code cases. Evaluated both raw and with the model's own template: generations are identical either way (4/12 both) — the template contributes nothing on this model.
Exact output tokens (own tokenizer) over the 4 solved: 26,017 = 6,504 tok/solved.
KL divergence vs the official inclusionAI/Ling-3.0-tiny-GGUF Q8_0 (ds4 short prompts, ctx 60, 2 chunks, CPU): per-chunk KL 0.00995 / 0.01375 (avg 0.0119), delta-p RMS 3.3-3.8%, same-token top-1 96.6 / 94.8%.
Eval parity with the Q4K-MXFP4-Q8 row (4/12 @ 124 t/s): this Q5_K_M build is a strictly higher-fidelity artifact of the same model (KL 0.0119, recipe-best class) at the same smoke score.

Downloads last month: 738

GGUF

Model size

8B params

Architecture

bailingmoe3

Hardware compatibility

4-bit

5-bit

Model tree for Ninnix96/Ling-3.0-tiny-gguf

Base model

inclusionAI/Ling-3.0-tiny

Quantized

(36)

this model

Collection including Ninnix96/Ling-3.0-tiny-gguf

Antirez's MoE recipe

Collection

Antirez recipe for MoE quantization: Q2 uses IQ2_XXS gate/up + Q2_K down; Q4 uses Q4_K gate/up + MXFP4 down. Everything else almost unchanged. • 9 items • Updated 2 days ago