Instructions to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Use Docker
docker model run hf.co/webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with Ollama:
ollama run hf.co/webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with Docker Model Runner:
docker model run hf.co/webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
- Lemonade
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Sakura-Llama-3.3-70B-Instruct-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Sakura ยท GGUF quantization of meta-llama/Llama-3.3-70B-Instruct
Independent community quantization. Not an official release of the upstream author.
Made with an importance matrix (imatrix). The weights were quantized with importance weights computed from a fixed calibration text; the exact method and the hashes are in the technical details of section 5.
1. Model Overview
Architecture: 70.6 B parameters ยท source formats: safetensors.
Original: 75.0 GB (Q8_0 GGUF, the source of these quantizations).
| Quant | Size | Compared with the original |
|---|---|---|
| Q4_K_M | 42.5 GB | 43 % smaller |
2. Original Model / Attribution
- Original model: meta-llama/Llama-3.3-70B-Instruct
- Declared base models: meta-llama/Llama-3.1-70B (relation:
finetune) - Quantized from:
bartowski/Llama-3.3-70B-Instruct-GGUF(fileLlama-3.3-70B-Instruct-Q8_0.gguf). - Quantized by
webmp3. Upstream authors retain their license and attribution.
3. Files
| File | Size | Quant |
|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
42.5 GB | Q4_K_M |
4. Download and Usage
llama-cli -m <file>.gguf -p "Hello" -n 32
Needs a llama.cpp build that supports this architecture. If a quantization is split into numbered parts, keep them together and load the first one.
5. Verification
- GGUF structure valid:
True - Real inference test (every file loaded by llama.cpp and used to generate text once):
yes - No coding benchmark, perplexity or quality score is claimed unless a measured value is listed here.
Technical details: exact versions, hashes, how these were made
Exact versions
- Original model commit:
6f6073b423013f6a7d4d9f39144961bfbfbc386b(model card at that commit) - Quantized from: commit
b6c5c9f176f3279204034e1d16d393105e95cb88, SHA-256d7314ee9cab0f263ec470fc984ceba44c0689645d6951f56a6e95f167383cc9c(provenance verified:True)
File hashes
| File | Bytes | SHA-256 | Quant |
|---|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
42520398880 | fa24f459f08a314d44d5e38df96a4a82083ab4c9500fd794dfbdf8ba2765cb6c |
Q4_K_M |
How these were made
This route performs existing-GGUF requantization from the pinned Q8_0 source GGUF named above. This job did not convert the upstream BF16/safetensors checkpoint.
Quantized using llama.cpp GGML kernels and an importance matrix (imatrix): the matrix was computed once with llama-imatrix from the fixed calibration text hfqa-original-calibration-v2 (6144 bytes of English, German and Chinese text, code and data formats, public domain; 12 chunks of 128 tokens) and applied during streaming with llama.cpp GGML quantization kernels. Matrix SHA-256: 47432094805c0279317c9abec0c50c897b3efbe4c493143e2816a124fac89e1e. Calibration text SHA-256: bc7216ea05c8bd05d6b9e94d813ec9eb651a2375288c08c06135a3f874d886f2. The calibration text is short; no quality improvement is claimed without a benchmark. Requantizing a previously quantized source can add quantization error.
The quant label names the llama.cpp preset. Stock llama.cpp keeps some tensors (for example normalization weights) in higher precision, so not every tensor is stored in the named type; the table shows what each file really contains.
| File | Tensors by type |
|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
F32: 162, Q4_K: 441, Q5_K: 40, Q6_K: 81 |
Test builds
| File | llama.cpp build | Executable SHA-256 |
|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
commit 19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4 |
323e0b05a9b3e7ed30851d4a393cf457e7a4372dbc0d1be2e93bc16235721fe9 |
6. License Information
Source license: llama3.3. license llama3.3: not checked automatically (operator decision); the card names the declared license and the original author's terms apply Built with Llama. This model is a derivative of Meta's Llama and is subject to the Llama community license; its text and acceptable use policy are in the original repository: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE
ไธญๆ่ฏดๆ ยท ๆจฑ่ฑ Sakura๏ผ็นๅปๅฑๅผ๏ผ
1. ๆฆ่ฟฐ
็ฌ็ซ็็คพๅบ้ๅ็ๆฌ๏ผๅนถ้ไธๆธธไฝ่ ็ๅฎๆนๅๅธใ
ไฝฟ็จ้่ฆๆง็ฉ้ต๏ผimatrix๏ผ้ๅใ ้ๅๆถไฝฟ็จไบ็ฑๅบๅฎๆ กๅๆๆฌ่ฎก็ฎๅบ็้่ฆๆงๆ้๏ผๅ ทไฝๆนๆณๅๅๅธ่ง็ฌฌ 5 ่็ๆๆฏ็ป่ใ
ๆถๆ๏ผ70.6 B ๅๆฐ ยท ๆบๆ ผๅผ๏ผsafetensorsใ
ๅๅงๆจกๅ๏ผ75.0 GB๏ผQ8_0 GGUF๏ผ่ฟไบ้ๅ็ๆฌ็ๆฅๆบ๏ผใ
| ้ๅ | ๅคงๅฐ | ็ธๆฏๅๅงๆจกๅ |
|---|---|---|
| Q4_K_M | 42.5 GB | ๅฐ 43 % |
2. ๅๅงๆจกๅไธ็ฝฒๅ
- ๅๅงๆจกๅ๏ผmeta-llama/Llama-3.3-70B-Instruct
- ๅฃฐๆ็ๅบ็กๆจกๅ๏ผmeta-llama/Llama-3.1-70B๏ผๅ
ณ็ณป๏ผ
finetune๏ผ - ้ๅๆฅๆบ๏ผ
bartowski/Llama-3.3-70B-Instruct-GGUF๏ผๆไปถLlama-3.3-70B-Instruct-Q8_0.gguf๏ผใ - ้ๅ่
๏ผ
webmp3ใไธๆธธไฝ่ ไฟ็ๅ ถ่ฎธๅฏ่ฏไธ็ฝฒๅใ
3. ๆไปถ
| ๆไปถ | ๅคงๅฐ | ้ๅ |
|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
42.5 GB | Q4_K_M |
4. ไธ่ฝฝไธไฝฟ็จ
llama-cli -m <file>.gguf -p "Hello" -n 32
้่ฆๆฏๆ่ฏฅๆถๆ็ llama.cpp ็ๆฌใๅฆๆๆไธช้ๅ่ขซๆๅไธบๅธฆ็ผๅท็ๅคไธช้จๅ๏ผ่ฏทๅฐๅฎไปฌๆพๅจไธ่ตทๅนถๅ ่ฝฝ็ฌฌไธไธชใ
5. ้ช่ฏ
- GGUF ็ปๆๆๆ๏ผ
True - ็ๅฎๆจ็ๆต่ฏ๏ผๆฏไธชๆไปถ้ฝ็ฑ llama.cpp ๅ ่ฝฝๅนถ็ๆไธๆฌกๆๆฌ๏ผ๏ผ
yes - ้ค้ๆญคๅคๅๅบไบๅฎๆตๆฐๅผ๏ผๅฆๅไธๅฃฐ็งฐไปปไฝ็ผ็จๅบๅใๅฐๆๅบฆๆ่ดจ้ๅๆฐใ
ๆๆฏ็ป่๏ผ็กฎๅ็ๆฌใๅๅธใๅถไฝๆนๅผ
็กฎๅ็ๆฌ
- ๅๅงๆจกๅๆไบค๏ผ
6f6073b423013f6a7d4d9f39144961bfbfbc386b๏ผ่ฏฅๆไบค็ๆจกๅๅก๏ผ - ้ๅๆฅๆบ๏ผๆไบค
b6c5c9f176f3279204034e1d16d393105e95cb88๏ผSHA-256d7314ee9cab0f263ec470fc984ceba44c0689645d6951f56a6e95f167383cc9c๏ผๅบๅคๅทฒ้ช่ฏ๏ผTrue๏ผ
ๆไปถๅๅธ
| ๆไปถ | ๅญ่ๆฐ | SHA-256 | ้ๅ |
|---|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
42520398880 | fa24f459f08a314d44d5e38df96a4a82083ab4c9500fd794dfbdf8ba2765cb6c |
Q4_K_M |
ๅถไฝๆนๅผ
่ฏฅ่ทฏ็บฟไปไธๆนๆ ๆ็ๅบๅฎ็ๆฌ Q8_0 ๆบ GGUF ่ฟ่ก็ฐๆ GGUF ็ๅ้ๅใๆฌไปปๅกๆฒกๆ่ฝฌๆขไธๆธธ็ BF16/safetensors ๆฃๆฅ็นใ
ไฝฟ็จ llama.cpp ๆ ๅๅทฅๅ
ทๅ้่ฆๆง็ฉ้ต๏ผimatrix๏ผ้ๅ๏ผ็ฉ้ต็ฑ llama-imatrix ๆ นๆฎๅบๅฎ็ๆ กๅๆๆฌ hfqa-original-calibration-v2๏ผ6144 ๅญ่๏ผๅซ่ฑๆใๅพทๆใไธญๆใไปฃ็ ๅๆฐๆฎๆ ผๅผ๏ผๅฑๅ
ฌๆ้ขๅ๏ผ12 ไธชๅ๏ผๆฏๅ 128 ไธช่ฏๅ
๏ผ่ฎก็ฎไธๆฌก๏ผๅจ้ๅๆตๅผ้ๅ่ฟ็จไธญ๏ผ็ฑ llama.cpp ็ GGML ้ๅๅ
ๆ ธๅบ็จใ็ฉ้ต SHA-256๏ผ47432094805c0279317c9abec0c50c897b3efbe4c493143e2816a124fac89e1eใๆ กๅๆๆฌ SHA-256๏ผbc7216ea05c8bd05d6b9e94d813ec9eb651a2375288c08c06135a3f874d886f2ใๆ กๅๆๆฌ่พ็ญ๏ผๆฒกๆๅบๅๆต่ฏ๏ผๅ ๆญคไธๅฃฐ็งฐ่ดจ้ๆๅใ ๅฏนๅทฒ้ๅ็ๆบๅๆฌก้ๅๅฏ่ฝๅขๅ ้ๅ่ฏฏๅทฎใ
้ๅๆ ็ญพๆ llama.cpp ็้ข่ฎพๅ็งฐใๆ ๅ llama.cpp ไผๆ้จๅๅผ ้๏ผไพๅฆๅฝไธๅๆ้๏ผไฟ็ไธบๆด้ซ็ฒพๅบฆ๏ผๅ ๆญคๅนถ้ๆฏไธชๅผ ้้ฝไปฅๆๆ ๆณจ็็ฑปๅๅญๅจ๏ผไธ่กจๅๅบๆฏไธชๆไปถๅฎ้ ๅ ๅซ็ๅ ๅฎนใ
| ๆไปถ | ๆ็ฑปๅ็ๅผ ้ๆฐ |
|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
F32: 162, Q4_K: 441, Q5_K: 40, Q6_K: 81 |
ๆต่ฏ็ๆฌ
| ๆไปถ | llama.cpp ็ๆฌ | ๅฏๆง่กๆไปถ SHA-256 |
|---|---|---|
Sakura-Llama-3.3-70B-Instruct-Q4_K_M.gguf |
commit 19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4 |
323e0b05a9b3e7ed30851d4a393cf457e7a4372dbc0d1be2e93bc16235721fe9 |
6. ่ฎธๅฏ่ฏไฟกๆฏ
ๆบ่ฎธๅฏ่ฏ๏ผllama3.3ใlicense llama3.3: not checked automatically (operator decision); the card names the declared license and the original author's terms apply
ๅฎๆด็ฌ็ซ็ไธญๆ็๏ผREADME_zh.md
- Downloads last month
- 172
4-bit
Model tree for webmp3/Sakura-Llama-3.3-70B-Instruct-GGUF
Base model
meta-llama/Llama-3.1-70B