zephyr-gguf / README.md
arunvpp05's picture
Update README.md
337df7a verified
|
Raw History Blame Contribute Delete
2.97 kB
metadata
language:
  - en
license: apache-2.0
datasets:
  - HuggingFaceH4/ultrachat_200k
  - Open-Orca/OpenOrca
  - OpenAssistant/oasst1
  - HuggingFaceH4/ultrafeedback_binarized
tags:
  - gguf
  - llama.cpp
  - quantized
  - local-inference
  - chat
pipeline_tag: text-generation
base_model: HuggingFaceH4/zephyr-7b-beta

Zephyr-7B-Beta β€” GGUF (Q5_K_M & Q8_0)

Two ready-to-run GGUF builds of the Zephyr-7B-Beta chat model for local CPU inference via the llama.cpp ecosystem.

These are inference-only quantized weights.


Files

  • zephyr-q5_k_m.gguf β€” balanced quality vs size (β‰ˆ 4.8 GB). Good default for 16 GB RAM laptops.
  • zephyr-q8_0.gguf β€” higher fidelity (β‰ˆ 7.2 GB). Requires more RAM.

GGUF embeds tokenizer/vocab, so separate tokenizer files are not required for inference.


Prompt Format (Zephyr chat)

Use Zephyr chat tags for best results:

<|user|> YOUR_PROMPT_HERE <|assistant|>

Example

<|user|> List three ways Retrieval-Augmented Generation improves factuality. <|assistant|>

How to Run (llama.cpp)

CLI (CPU)

Q5_K_M (fits most 16 GB RAM systems)

./llama-cli -m zephyr-q5_k_m.gguf \
  -p "<|user|>\nExplain RAG in 3 bullets.\n\n<|assistant|>\n" \
  -n 256 -c 2048 -ngl 0 -t $(nproc)

Q8_0 (higher quality; more RAM)

./llama-cli -m zephyr-q8_0.gguf \
  -p "<|user|>\nGive 5 note-taking tips.\n\n<|assistant|>\n" \
  -n 256 -c 2048 -ngl 0 -t $(nproc)

Flags

  • -n 256 β†’ max new tokens
  • -c 2048 β†’ context window
  • -ngl 0 β†’ CPU-only (set >0 to offload to GPU if supported)
  • -t $(nproc) β†’ threads

Some builds use ./main instead of ./llama-cli. Replace the binary name if needed.

Popular UIs

Import the .gguf directly in:

  • LM Studio
  • KoboldCpp
  • Text Generation WebUI (llama.cpp backend)
  • Ollama (custom import)

Hardware Notes

Approximate RAM use at 2k context (CPU-only):

  • Q5_K_M (~4.8 GB file) β†’ ~8–10 GB RAM
  • Q8_0 (~7.2 GB file) β†’ ~12–14 GB RAM

Actual usage varies with context length, batch size, and compile options.


Checksums (optional)

Verify downloads:

sha256sum zephyr-q5_k_m.gguf
sha256sum zephyr-q8_0.gguf

(Add the resulting hashes here if you want to publish them.)


Intended Use & Limitations

  • Intended for local assistant/chat and general text generation.
  • Not suitable for high-stakes or safety-critical use without human review.
  • Outputs may contain mistakes or biases; verify important information.

What’s Included

Quantized GGUF weights:

  • zephyr-q5_k_m.gguf
  • zephyr-q8_0.gguf

No training code or LoRA adapters are included here.


Acknowledgments

  • Base: Zephyr-7B-Beta (converted to GGUF and quantized for CPU inference).
  • Inference runtime: llama.cpp and compatible UIs.

Changelog

  • v1.0 β€” Initial release of Q5_K_M and Q8_0 GGUF builds.