kingabzpro commited on
Commit
3873197
·
verified ·
1 Parent(s): 1693888

Deploy llama.cpp web UI with Qwythos-9B MTP model

Browse files
Files changed (3) hide show
  1. Dockerfile +35 -0
  2. README.md +35 -5
  3. start.sh +25 -0
Dockerfile ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # llama.cpp web UI on a Hugging Face Docker Space
2
+ # CUDA runtime base so the prebuilt CUDA build of llama.cpp can run on the A10G.
3
+ FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04
4
+
5
+ ENV DEBIAN_FRONTEND=noninteractive
6
+
7
+ # curl + zstd are required by the llama.app installer; ca-certificates for HTTPS.
8
+ RUN apt-get update \
9
+ && apt-get install -y --no-install-recommends \
10
+ curl \
11
+ ca-certificates \
12
+ zstd \
13
+ && rm -rf /var/lib/apt/lists/*
14
+
15
+ # Hugging Face Spaces run as uid 1000; create a matching non-root user.
16
+ RUN useradd --create-home --uid 1000 --shell /bin/bash user
17
+
18
+ WORKDIR /home/user/app
19
+
20
+ # --- Bake the GGUF model into the image (build-time, ~5.9 GB) -----------------
21
+ # Public model, no token required. Placed in its own layer so editing start.sh
22
+ # below does not bust this cache.
23
+ USER user
24
+ ARG MODEL_REPO=empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF
25
+ ARG MODEL_FILE=Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf
26
+ RUN curl -fL --retry 3 --retry-delay 5 \
27
+ -o "/home/user/app/${MODEL_FILE}" \
28
+ "https://huggingface.co/${MODEL_REPO}/resolve/main/${MODEL_FILE}"
29
+
30
+ # --- Startup script -----------------------------------------------------------
31
+ COPY --chown=user:user start.sh /home/user/app/start.sh
32
+ RUN chmod +x /home/user/app/start.sh
33
+
34
+ EXPOSE 8080
35
+ CMD ["/home/user/app/start.sh"]
README.md CHANGED
@@ -1,10 +1,40 @@
1
  ---
2
- title: Llamacpp Webui
3
- emoji: 📉
4
- colorFrom: blue
5
- colorTo: green
6
  sdk: docker
 
7
  pinned: false
 
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: LlamaCpp WebUI
3
+ emoji: "\U0001F999"
4
+ colorFrom: indigo
5
+ colorTo: blue
6
  sdk: docker
7
+ app_port: 8080
8
  pinned: false
9
+ hardware: a10g-large
10
+ models:
11
+ - empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF
12
  ---
13
 
14
+ # llama.cpp WebUI (llama-server)
15
+
16
+ Runs the official **llama.cpp** `llama serve` web UI on a Hugging Face Space (Docker SDK, NVIDIA A10G).
17
+
18
+ ## Model
19
+ [`empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF`](https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF)
20
+ Quantization: `Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf` (MTP variant for speculative decoding).
21
+
22
+ ## How it works
23
+ - The prebuilt `llama.cpp` binary is installed at **runtime** via the official one-liner
24
+ `curl -LsSf https://llama.app/install.sh | sh`. The installer probes the hardware and
25
+ downloads the CUDA build (an A10G is present at runtime).
26
+ - The GGUF model is baked into the image at **build time** for instant cold starts.
27
+ - `llama serve` is launched with Multi-Token-Prediction speculative decoding:
28
+
29
+ ```bash
30
+ llama serve \
31
+ -m Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf \
32
+ --spec-type draft-mtp \
33
+ --spec-draft-n-max 6 \
34
+ -c 16384 \
35
+ --host 0.0.0.0 \
36
+ --port 8080 \
37
+ -ngl 99
38
+ ```
39
+
40
+ The web UI is served on the Space app port `8080`.
start.sh ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+
4
+ export PATH="$HOME/.local/bin:${PATH}"
5
+
6
+ # Install the prebuilt llama.cpp binary at runtime. The installer probes the
7
+ # hardware and downloads the CUDA build because a GPU is available here.
8
+ if ! command -v llama >/dev/null 2>&1; then
9
+ echo "==> Installing prebuilt llama.cpp via llama.app ..."
10
+ curl -LsSf https://llama.app/install.sh | sh
11
+ fi
12
+
13
+ cd /home/user/app
14
+
15
+ MODEL_FILE="Qwythos-9B-Claude-Mythos-5-1M-MTP-Q4_K_M.gguf"
16
+
17
+ echo "==> Launching llama serve (web UI on :8080) ..."
18
+ exec llama serve \
19
+ -m "${MODEL_FILE}" \
20
+ --spec-type draft-mtp \
21
+ --spec-draft-n-max 6 \
22
+ -c 16384 \
23
+ --host 0.0.0.0 \
24
+ --port 8080 \
25
+ -ngl 99