Spaces:
Paused
Paused
Deploy llama.cpp web UI with Qwythos-9B MTP model
Browse files- Dockerfile +35 -0
- README.md +35 -5
- start.sh +25 -0
Dockerfile
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# llama.cpp web UI on a Hugging Face Docker Space
|
| 2 |
+
# CUDA runtime base so the prebuilt CUDA build of llama.cpp can run on the A10G.
|
| 3 |
+
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
|
| 4 |
+
|
| 5 |
+
ENV DEBIAN_FRONTEND=noninteractive
|
| 6 |
+
|
| 7 |
+
# curl + zstd are required by the llama.app installer; ca-certificates for HTTPS.
|
| 8 |
+
RUN apt-get update \
|
| 9 |
+
&& apt-get install -y --no-install-recommends \
|
| 10 |
+
curl \
|
| 11 |
+
ca-certificates \
|
| 12 |
+
zstd \
|
| 13 |
+
&& rm -rf /var/lib/apt/lists/*
|
| 14 |
+
|
| 15 |
+
# Hugging Face Spaces run as uid 1000; create a matching non-root user.
|
| 16 |
+
RUN useradd --create-home --uid 1000 --shell /bin/bash user
|
| 17 |
+
|
| 18 |
+
WORKDIR /home/user/app
|
| 19 |
+
|
| 20 |
+
# --- Bake the GGUF model into the image (build-time, ~5.9 GB) -----------------
|
| 21 |
+
# Public model, no token required. Placed in its own layer so editing start.sh
|
| 22 |
+
# below does not bust this cache.
|
| 23 |
+
USER user
|
| 24 |
+
ARG MODEL_REPO=empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF
|
| 25 |
+
ARG MODEL_FILE=Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf
|
| 26 |
+
RUN curl -fL --retry 3 --retry-delay 5 \
|
| 27 |
+
-o "/home/user/app/${MODEL_FILE}" \
|
| 28 |
+
"https://huggingface.co/${MODEL_REPO}/resolve/main/${MODEL_FILE}"
|
| 29 |
+
|
| 30 |
+
# --- Startup script -----------------------------------------------------------
|
| 31 |
+
COPY --chown=user:user start.sh /home/user/app/start.sh
|
| 32 |
+
RUN chmod +x /home/user/app/start.sh
|
| 33 |
+
|
| 34 |
+
EXPOSE 8080
|
| 35 |
+
CMD ["/home/user/app/start.sh"]
|
README.md
CHANGED
|
@@ -1,10 +1,40 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: docker
|
|
|
|
| 7 |
pinned: false
|
|
|
|
|
|
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: LlamaCpp WebUI
|
| 3 |
+
emoji: "\U0001F999"
|
| 4 |
+
colorFrom: indigo
|
| 5 |
+
colorTo: blue
|
| 6 |
sdk: docker
|
| 7 |
+
app_port: 8080
|
| 8 |
pinned: false
|
| 9 |
+
hardware: a10g-large
|
| 10 |
+
models:
|
| 11 |
+
- empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF
|
| 12 |
---
|
| 13 |
|
| 14 |
+
# llama.cpp WebUI (llama-server)
|
| 15 |
+
|
| 16 |
+
Runs the official **llama.cpp** `llama serve` web UI on a Hugging Face Space (Docker SDK, NVIDIA A10G).
|
| 17 |
+
|
| 18 |
+
## Model
|
| 19 |
+
[`empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF`](https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF)
|
| 20 |
+
Quantization: `Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf` (MTP variant for speculative decoding).
|
| 21 |
+
|
| 22 |
+
## How it works
|
| 23 |
+
- The prebuilt `llama.cpp` binary is installed at **runtime** via the official one-liner
|
| 24 |
+
`curl -LsSf https://llama.app/install.sh | sh`. The installer probes the hardware and
|
| 25 |
+
downloads the CUDA build (an A10G is present at runtime).
|
| 26 |
+
- The GGUF model is baked into the image at **build time** for instant cold starts.
|
| 27 |
+
- `llama serve` is launched with Multi-Token-Prediction speculative decoding:
|
| 28 |
+
|
| 29 |
+
```bash
|
| 30 |
+
llama serve \
|
| 31 |
+
-m Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf \
|
| 32 |
+
--spec-type draft-mtp \
|
| 33 |
+
--spec-draft-n-max 6 \
|
| 34 |
+
-c 16384 \
|
| 35 |
+
--host 0.0.0.0 \
|
| 36 |
+
--port 8080 \
|
| 37 |
+
-ngl 99
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
The web UI is served on the Space app port `8080`.
|
start.sh
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
|
| 4 |
+
export PATH="$HOME/.local/bin:${PATH}"
|
| 5 |
+
|
| 6 |
+
# Install the prebuilt llama.cpp binary at runtime. The installer probes the
|
| 7 |
+
# hardware and downloads the CUDA build because a GPU is available here.
|
| 8 |
+
if ! command -v llama >/dev/null 2>&1; then
|
| 9 |
+
echo "==> Installing prebuilt llama.cpp via llama.app ..."
|
| 10 |
+
curl -LsSf https://llama.app/install.sh | sh
|
| 11 |
+
fi
|
| 12 |
+
|
| 13 |
+
cd /home/user/app
|
| 14 |
+
|
| 15 |
+
MODEL_FILE="Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf"
|
| 16 |
+
|
| 17 |
+
echo "==> Launching llama serve (web UI on :8080) ..."
|
| 18 |
+
exec llama serve \
|
| 19 |
+
-m "${MODEL_FILE}" \
|
| 20 |
+
--spec-type draft-mtp \
|
| 21 |
+
--spec-draft-n-max 6 \
|
| 22 |
+
-c 16384 \
|
| 23 |
+
--host 0.0.0.0 \
|
| 24 |
+
--port 8080 \
|
| 25 |
+
-ngl 99
|