Spaces:
Sleeping
Sleeping
Commit History
-
docs: temporary pause notice / personal mirror pointer
Verified signature111c86e
-
widen the throughput claim to the measured range (~150-260 tok/s)
Verified signatureThe 190-260 figure came from a single run. A second run an hour later gave 148-182. cpu-basic is shared, so the honest claim is a range: ~150-260 tok/s, ~14-23x the RV1103.e209677 -
correct the throughput claim: ~230 tok/s sustained (~21x), not 1600 (146x)
Verified signatureThe 1604 tok/s / 146x figure did not come from this hardware. Measured on the running free cpu-basic Space: 190-260 tok/s sustained, ~21x the RV1103's 11 tok/s. Also records that the first call in a fresh container is 7-16 tok/s -- at or below the device -- because the lazy model load is inside the timed region, and the Space sleeps when idle.2f17677 -
size llama.cpp threads from the cgroup quota, not os.cpu_count()
Verified signatureos.cpu_count() reports the host's 16 cores; a free cpu-basic container is limited to ~2 by its cgroup. Oversubscribing ~8x made a 16M model decode at 0.5-1.2 tok/s -- slower than the RV1103 this endpoint exists to offload.11a61b3 -
install llama-cpp-python prebuilt from conda-forge instead of compiling it
Verified signatureCompiling exceeds the Docker build timeout even at -j16 with tests, examples, server and tools disabled (two attempts, both Job timeout). conda-forge publishes a prebuilt linux-64 cp311 binary built against a glibc sysroot: correct ABI, and no compile, so build time stops mattering.5a811ec -
parallelise + narrow the llama.cpp compile (Docker build also hits Job timeout)
Verified signatureA Docker build is not exempt from the Space build timeout. cmake builds serially by default, so pass CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) and use Ninja. Also drop build scope the Space never loads: tests, examples, the CLI/server tools and curl. ISA flags unchanged -- still only GGML_NATIVE=OFF.beb2674 -
convert to sdk:docker -- build llama-cpp-python against this image's glibc
Verified signatureThe abetlen wheel index ships bare linux_x86_64 wheels built on Alpine; libllama.so links musl and cannot dlopen on the Debian/glibc Space image, so /v1/chat/completions 500'd on every call. PyPI has sdist only, which times out the Spaces pip step. Compile in a bookworm builder stage (gcc 12, GGML_NATIVE=OFF), install the resulting wheel in a matching bookworm runtime with libgomp1. Endpoints and the 7860 binding are unchanged; still free cpu-basic.fade774 -
bind GRADIO_SERVER_PORT (7860), not PORT (7861, already bound)
Verified signatured15fa32 -
serve the ASGI app with uvicorn -- defining `app` never served it (exit 0)
Verified signature2b904e4 -
pin python_version 3.11 for llama-cpp-python cp311 wheels
Verified signature8c39881 -
llama-cpp-python 0.3.19 from the CPU wheel index (0.3.16 had no wheel -> source build timeout)
Verified signature01a1cda -
Upload folder using huggingface_hub
Verified signatureee9cf5a -
initial commit
Verified signaturef3fac53