Instructions to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Use Docker
docker model run hf.co/webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with Ollama:
ollama run hf.co/webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with Docker Model Runner:
docker model run hf.co/webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
- Lemonade
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Sakura-Qwen2.5-0.5B-Instruct-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "webmp3/Sakura-Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Sakura ยท GGUF quantization of Qwen/Qwen2.5-0.5B-Instruct
- 1. Model Overview
- 2. Original Model / Attribution
- 3. Source Revision
- 4. Supported GGUF Quantizations
- 5. Quantization Methodology
- 6. Architecture Information
- 7. Conversion and Runtime Compatibility
- 8. Download and Usage Instructions
- 9. Validation Results
- 10. Limitations
- 11. License Information
- 12. Reproduction Information
- 1. Model Overview
Sakura ยท GGUF quantization of Qwen/Qwen2.5-0.5B-Instruct
Independent community quantization. Not an official release of the upstream author.
1. Model Overview
This repository is produced by the local HF Quant Agent control plane. It records only checks that actually ran. Unmeasured quality, speed and compatibility are marked untested.
2. Original Model / Attribution
- Source: Qwen/Qwen2.5-0.5B-Instruct
- Publisher of this quantization:
webmp3, when a release is actually uploaded - Upstream authors retain their license and attribution
- GGUF source publisher:
Qwen/Qwen2.5-0.5B-Instruct-GGUF - GGUF source commit:
9217f5db79a29953eb74d5343926648285ec7e67 - GGUF source filename:
qwen2.5-0.5b-instruct-q8_0.gguf - GGUF source SHA-256:
ca59ca7f13d0e15a8cfa77bd17e65d24f6844b554a7b6c12e07a5f89ff76844e - GGUF source provenance verified:
True
3. Source Revision
Pinned revision: 7ae557604adf67be50417f59c2c2f167def9a775
Jobs are bound to this SHA. A later upstream commit does not silently change a running or completed job.
4. Supported GGUF Quantizations
Requested: Q3_K_M
| File | Bytes | SHA-256 | Quant |
|---|---|---|---|
Sakura-qwen2.5-0.5b-instruct-q8_0-Q3_K_M-requant-00001-of-00001.gguf |
432041920 | 1a76ce5a1cda6afe188ebbc5de47097890c48539aa4b44e4f5fc87f72117365d |
Q3_K_M |
5. Quantization Methodology
Route: native_gguf_stream
Adapter: standard-llamacpp
This route performs existing-GGUF requantization from the pinned Q8_0/F16/BF16 source GGUF identified above. This job did not convert the upstream BF16/safetensors checkpoint. Source tensor storage measured in the GGUF: F32: 121, Q8_0: 170.
No importance-matrix calibration was run by this job. Requantizing a previously quantized source can add quantization error. Estimated sizes are planning ratios, not measurements, unless a file byte count is listed above.
The target quant label describes the selected stock quantizer preset. Stock llama.cpp may retain floating-point tensors or use fallback tensor types; every tensor is not necessarily stored in the named target type. The table records measured output tensor types and only reports a fallback count when the quantizer printed one.
Source tensor count: 291. Source full tensor-name set SHA-256: 70e194dd80c4c853c75f2073b7209f1ca5c93265d2ed7bd1836f82ac67bcc8b0. New and reused outputs must retain the source tensor count and complete name set.
| File | Measured tensor-type counts | Quantizer-reported fallback tensors | Full tensor-name set SHA-256 |
|---|---|---|---|
Sakura-qwen2.5-0.5b-instruct-q8_0-Q3_K_M-requant-00001-of-00001.gguf |
F32: 121, Q4_0: 97, Q4_K: 23, Q5_0: 46, Q5_1: 2, Q5_K: 1, Q8_0: 1 | unmeasured; inspect quantizer log | 70e194dd80c4c853c75f2073b7209f1ca5c93265d2ed7bd1836f82ac67bcc8b0 |
6. Architecture Information
- model_type / architecture:
qwen2 - parameters:
494032768 - experts:
n/a or not declared - source formats observed: safetensors, bfloat16
- custom code present:
False trust_remote_codeexecuted:False
7. Conversion and Runtime Compatibility
- GGUF structure valid:
True - Runtime load tested:
True - Inference tested:
True
| File | Runtime build reported by executable | Runtime executable SHA-256 |
|---|---|---|
Sakura-qwen2.5-0.5b-instruct-q8_0-Q3_K_M-requant-00001-of-00001.gguf |
b11669-620d2225c |
486568ae9dd8682e160861a129b70f4a5efa73191542ab0cf80674b8b6d4f814 |
8. Download and Usage Instructions
Use a llama.cpp build that supports this architecture. Example, after you confirm compatibility:
llama-cli -m <file>.gguf -p "Hello" -n 32
This card does not claim that every runtime can load the file.
9. Validation Results
All selected streamed output compositions passed structural checks and one stock CPU inference per quant. Every composed shard was hashed before and after inference.
10. Limitations
- No coding benchmark, perplexity or quality score is claimed unless a measured value is copied from this job's own log.
- Shard resume and upload verification apply only to artifacts listed in this card.
11. License Information
Source license: apache-2.0
Decision: apache-2.0 allows redistribution with attribution
If the license does not allow redistribution, this card must not be uploaded.
12. Reproduction Information
- Control plane: HF Quant Agent
- Planner route:
native_gguf_stream - Workflow:
none - Blockers: none recorded
- Downloads last month
- 407
2-bit
3-bit
4-bit
5-bit