--- language: - "en" license: "apache-2.0" datasets: - "HuggingFaceH4/ultrachat_200k" - "Open-Orca/OpenOrca" - "OpenAssistant/oasst1" - "HuggingFaceH4/ultrafeedback_binarized" tags: - "gguf" - "llama.cpp" - "quantized" - "local-inference" - "chat" pipeline_tag: "text-generation" base_model: "HuggingFaceH4/zephyr-7b-beta" --- # Zephyr-7B-Beta — GGUF (Q5_K_M & Q8_0) Two ready-to-run **GGUF** builds of the Zephyr-7B-Beta chat model for local CPU inference via the `llama.cpp` ecosystem. These are **inference-only** quantized weights. --- ## Files - `zephyr-q5_k_m.gguf` — balanced quality vs size (≈ 4.8 GB). Good default for 16 GB RAM laptops. - `zephyr-q8_0.gguf` — higher fidelity (≈ 7.2 GB). Requires more RAM. > GGUF embeds tokenizer/vocab, so separate tokenizer files are not required for inference. --- ## Prompt Format (Zephyr chat) Use Zephyr chat tags for best results: <|user|> YOUR_PROMPT_HERE <|assistant|> **Example** <|user|> List three ways Retrieval-Augmented Generation improves factuality. <|assistant|> --- ## How to Run (llama.cpp) ### CLI (CPU) **Q5_K_M (fits most 16 GB RAM systems)** ./llama-cli -m zephyr-q5_k_m.gguf \ -p "<|user|>\nExplain RAG in 3 bullets.\n\n<|assistant|>\n" \ -n 256 -c 2048 -ngl 0 -t $(nproc) **Q8_0 (higher quality; more RAM)** ./llama-cli -m zephyr-q8_0.gguf \ -p "<|user|>\nGive 5 note-taking tips.\n\n<|assistant|>\n" \ -n 256 -c 2048 -ngl 0 -t $(nproc) *Flags* - `-n 256` → max new tokens - `-c 2048` → context window - `-ngl 0` → CPU-only (set `>0` to offload to GPU if supported) - `-t $(nproc)` → threads > Some builds use `./main` instead of `./llama-cli`. Replace the binary name if needed. ### Popular UIs Import the `.gguf` directly in: - LM Studio - KoboldCpp - Text Generation WebUI (llama.cpp backend) - Ollama (custom import) --- ## Hardware Notes Approximate RAM use at 2k context (CPU-only): - **Q5_K_M (~4.8 GB file)** → ~8–10 GB RAM - **Q8_0 (~7.2 GB file)** → ~12–14 GB RAM Actual usage varies with context length, batch size, and compile options. --- ## Checksums (optional) Verify downloads: sha256sum zephyr-q5_k_m.gguf sha256sum zephyr-q8_0.gguf (Add the resulting hashes here if you want to publish them.) --- ## Intended Use & Limitations - Intended for local assistant/chat and general text generation. - Not suitable for high-stakes or safety-critical use without human review. - Outputs may contain mistakes or biases; verify important information. --- ## What’s Included Quantized GGUF weights: - `zephyr-q5_k_m.gguf` - `zephyr-q8_0.gguf` No training code or LoRA adapters are included here. --- ## Acknowledgments - Base: Zephyr-7B-Beta (converted to GGUF and quantized for CPU inference). - Inference runtime: `llama.cpp` and compatible UIs. --- ## Changelog - **v1.0** — Initial release of `Q5_K_M` and `Q8_0` GGUF builds.