GLM-5.3-Flash FP4 HYPERFLARE β 1M, TP2 (two RTX PRO 6000)
A prepared Hyperflare runtime
package for GLM-5.3-Flash on two RTX PRO 6000 Blackwell cards
(GB202 / SM120, 96 GB each), tensor-parallel TP2. The base checkpoint is
nvidia/GLM-5.3-Flash-FP4-Local
at 76348ddb β the checkpoint the in-tree GLM recipe qualifies β served with
vision, MTP speculation, and the full 1,048,576-token context.
Credit: This repository packages and publishes upstream work; it does not claim authorship of the model, runtime, recipes, kernels, or the GLM fixes. See Credits and provenance.
The package payload lives on the gb202 revision. This main branch carries
metadata only. Hyperflare selects gb202 automatically on GB202 devices; the
explicit form is MODEL@gb202.
Required Hyperflare version
The package uses execution ABI 6 / prepared model contract 3. Build
Hyperflare from
NVIDIA/hyperflare-inference
at main commit 68bbd6dd (2026-11-08) or newer:
git clone https://github.com/NVIDIA/hyperflare-inference
cd hyperflare-inference
git checkout 68bbd6dd
cargo build --release --manifest-path flare/Cargo.toml
The project was renamed in the README at d2b5c5f3 (documentation-only); the
GitHub repository, formerly spark-local-inference, now redirects to
hyperflare-inference, and the binary is still named flare. Older builds (before execution ABI 6) reject this package
before replay. Serving needs no Python; preparation-only dependencies
(FlashInfer, liburing headers, CUDA toolkit) are not required to serve. Replay
requires the same GPU architecture the package was prepared on: GB202
(RTX PRO 6000 Blackwell, SM120), exactly two ranks.
Use
flare run rickybrent/GLM-5.3-Flash-HYPERFLARE-TP2 --gpus 0,1
For a resumable Xet download (helpful for the 180 GB payload), fetch the file first and run it locally:
hf download rickybrent/GLM-5.3-Flash-HYPERFLARE-TP2 \
glm-5.3-flash-fp4-hyperflare-1m-tp2.safetensors --revision gb202 \
--local-dir glm53-tp2
flare run glm53-tp2/glm-5.3-flash-fp4-hyperflare-1m-tp2.safetensors --gpus 0,1
--gpus must select exactly 2 devices. On first load Hyperflare shows the publisher
disclosure (license terms and signing-key fingerprint); type rickybrent to
accept. For unattended startup:
--dangerously-approve-key 5deb049111540799f1bf1da1eb1b912f1015313d7a10b743318d70cd5e0ed45e \
--dangerously-accept-publisher rickybrent
Identity verification fetches publisher.json from this
repository's root on every online load. This is a production signature
(development: false), not a development key.
Example request (OpenAI-compatible API):
curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "GLM-5.3-Flash",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 200
}'
Package facts
| Property | Value |
|---|---|
| Prepared model contract / execution ABI | 3 / 6 |
| Checkpoint | nvidia/GLM-5.3-Flash-FP4-Local @ 76348ddb (~165 GiB downloaded during preparation) |
| Hardware | 2 Γ RTX PRO 6000 Blackwell (GB202, SM120, 96 GB), TP2 |
| Serving context | 1,048,576 tokens per request |
| Speculation | MTP, 5 draft tokens default, widths 1β6, adaptive verification |
| Prepared capacities | concurrency 4, prefill chunk 4096 |
| KV cache at load | 1,405,952 tokens Β· 54 shared snapshot slots Β· 9.6 GiB total |
| Packed package size | ~180.3 GB |
| Loaded per GPU | 88.94β88.97 GiB Β· 4.82β4.88 GiB cache budget |
| Captured CUDA graphs / kernels | 71 / 5,664 |
| Cold load | ~43 s (35.7 s reading the package) |
Performance and validation
Measured 2026-11-08 on the two RTX PRO 6000s with the LIL bench, 30 s cells,
c1/2/4 at 0/16K/128K contexts. The package was prepared from local branch
glm53-prs (d2b5c5f3 plus preparation PRs #217, #218 and #231; those PRs are
unmerged and preparation-only, so serving does not need the branch):
| Prefill | 8K | 16K | 64K | 128K |
|---|---|---|---|---|
| tok/s | 3,965 | 4,094 | 4,160 | 4,019 |
| Decode, aggregate tok/s | C1 | C2 | C4 |
|---|---|---|---|
| 0K | 185.7 | 319.4 | 364.8 |
| 16K | 184.6 | 307.3 | 377.2 |
| 128K | 194.0 | 293.5 | 386.1 |
Against the same package prepared from upstream 68bbd6dd without the PRs,
C2 improved +15.0% / +10.2% / +2.8% at 0K/16K/128K; C1 moved
+0.6% / +1.1% / +8.1% and C4 β2.4% / β2.6% / +2.9%. The 0K C2 gain occurs with
lower acceptance length (2.152 β 1.813, accept rate 0.705 β 0.769), i.e. faster
rounds rather than sampling luck. Raw runs are under bench/.
Validation on this package:
- cached speculative battery: 3 correct + 1 neighbour, 0 corrupt
- cold 70K Γ3: 3/3 correct; 350K and ~960K retrieval correct
- 36 text probes: 0
[gMASK], 0 budget starvation - tool calls 6/6; image test correct
- loads through
flare runwith the production publisher and 1M context
Reproduction
git clone https://github.com/NVIDIA/hyperflare-inference && cd hyperflare-inference
git checkout d2b5c5f3
git fetch origin pull/217/head:pr-217 pull/218/head:pr-218 pull/231/head:pr-231
git switch -c glm53-prs && git merge --no-edit pr-217 pr-218 pr-231
uv sync --locked --extra prepare --extra test
cargo build --release --manifest-path flare/Cargo.toml
uv pip install --python .venv/bin/python --no-deps ./heliopause
PATH=$PWD/.venv/bin:$PATH .venv/bin/ignition export glm-5.3-flash \
--gpus 0,1 --max-concurrency 4
.venv/bin/ignition convert CACHE_ENTRY \
--form packed --output glm53-tp2-unsigned.safetensors
--reuse-weights is optional when a compatible cached default exists. Promote
with ignition sign under your own publisher identity before publishing;
serving the package needs only flare run, not the preparation environment.
Caveats
- GB202/SM120, TP2, concurrency 4 and a 4096-token prefill chunk are baked into the package. Re-export for other architectures, TP sizes or capacities.
- 1,048,576 tokens is the per-request context; the 1,405,952-token pool is shared by up to four concurrent requests, so several full-context conversations cannot be resident at once.
- The PR package was prepared from unmerged preparation PRs. Every token is verified by the target model, and the correctness battery above is clean; the mainline package remains the conservative reproduction path.
- The package keeps both cards almost full (~93.8 GiB each, about 98% of usable memory) while loaded. Leave the box otherwise idle, and stop the server with SIGTERM so native replay drains before CUDA teardown.
Credits and provenance
This repository is a packaging, validation and publication effort only. The model, runtime, recipes, kernels, benchmark tooling and the GLM fixes are the work of others. No authorship claim is made over any upstream work.
- Local Inference Lab and the NVIDIA Hyperflare (formerly Flare) authors and contributors β runtime, recipes, package format and documentation (
NVIDIA/hyperflare-inference). - NVIDIA and Z.ai β the GLM-5.3-Flash FP4 checkpoint (
nvidia/GLM-5.3-Flash-FP4-Local) and the GLM-5.3-Flash base model. - Jason Cook (original-el8) β PR #129, the preparation copy-ordering fix that makes GLM speculative decoding correct, and PRs #217/#218/#231, the preparation improvements baked into this package.
- Local Inference Lab β FlashInfer,
llm-inference-bench, the b12x mirror (b12x is Luke Alonso's, with Martin Vit and contributors), and the LIL community whose testing and discussion made this bring-up possible. - JohannesBertens and JMNI-Labs β prior Flare package layouts this repository follows.
Packaged, validated and published on a host with two RTX PRO 6000 GPUs. The package's publisher identity is a signing identity, not an authorship claim. All credit for the model, engine, kernels and fixes belongs to the projects and people listed above.
License
Weights derive from
nvidia/GLM-5.3-Flash-FP4-Local
and are subject to the NVIDIA Open Model Agreement (license recorded by the
exporter: nvidia-open-model-agreement). Review and accept that agreement
before use. Hyperflare is Apache-2.0.
- Downloads last month
- 87