GLM-5.3-Flash FP4 HYPERFLARE β€” 1M, TP2 (two RTX PRO 6000)

A prepared Hyperflare runtime package for GLM-5.3-Flash on two RTX PRO 6000 Blackwell cards (GB202 / SM120, 96 GB each), tensor-parallel TP2. The base checkpoint is nvidia/GLM-5.3-Flash-FP4-Local at 76348ddb β€” the checkpoint the in-tree GLM recipe qualifies β€” served with vision, MTP speculation, and the full 1,048,576-token context.

Credit: This repository packages and publishes upstream work; it does not claim authorship of the model, runtime, recipes, kernels, or the GLM fixes. See Credits and provenance.

The package payload lives on the gb202 revision. This main branch carries metadata only. Hyperflare selects gb202 automatically on GB202 devices; the explicit form is MODEL@gb202.

Required Hyperflare version

The package uses execution ABI 6 / prepared model contract 3. Build Hyperflare from NVIDIA/hyperflare-inference at main commit 68bbd6dd (2026-11-08) or newer:

git clone https://github.com/NVIDIA/hyperflare-inference
cd hyperflare-inference
git checkout 68bbd6dd
cargo build --release --manifest-path flare/Cargo.toml

The project was renamed in the README at d2b5c5f3 (documentation-only); the GitHub repository, formerly spark-local-inference, now redirects to hyperflare-inference, and the binary is still named flare. Older builds (before execution ABI 6) reject this package before replay. Serving needs no Python; preparation-only dependencies (FlashInfer, liburing headers, CUDA toolkit) are not required to serve. Replay requires the same GPU architecture the package was prepared on: GB202 (RTX PRO 6000 Blackwell, SM120), exactly two ranks.

Use

flare run rickybrent/GLM-5.3-Flash-HYPERFLARE-TP2 --gpus 0,1

For a resumable Xet download (helpful for the 180 GB payload), fetch the file first and run it locally:

hf download rickybrent/GLM-5.3-Flash-HYPERFLARE-TP2 \
  glm-5.3-flash-fp4-hyperflare-1m-tp2.safetensors --revision gb202 \
  --local-dir glm53-tp2
flare run glm53-tp2/glm-5.3-flash-fp4-hyperflare-1m-tp2.safetensors --gpus 0,1

--gpus must select exactly 2 devices. On first load Hyperflare shows the publisher disclosure (license terms and signing-key fingerprint); type rickybrent to accept. For unattended startup:

  --dangerously-approve-key 5deb049111540799f1bf1da1eb1b912f1015313d7a10b743318d70cd5e0ed45e \
  --dangerously-accept-publisher rickybrent

Identity verification fetches publisher.json from this repository's root on every online load. This is a production signature (development: false), not a development key.

Example request (OpenAI-compatible API):

curl -s http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "GLM-5.3-Flash",
  "messages": [{"role": "user", "content": "Hello"}],
  "max_tokens": 200
}'

Package facts

Property Value
Prepared model contract / execution ABI 3 / 6
Checkpoint nvidia/GLM-5.3-Flash-FP4-Local @ 76348ddb (~165 GiB downloaded during preparation)
Hardware 2 Γ— RTX PRO 6000 Blackwell (GB202, SM120, 96 GB), TP2
Serving context 1,048,576 tokens per request
Speculation MTP, 5 draft tokens default, widths 1–6, adaptive verification
Prepared capacities concurrency 4, prefill chunk 4096
KV cache at load 1,405,952 tokens Β· 54 shared snapshot slots Β· 9.6 GiB total
Packed package size ~180.3 GB
Loaded per GPU 88.94–88.97 GiB Β· 4.82–4.88 GiB cache budget
Captured CUDA graphs / kernels 71 / 5,664
Cold load ~43 s (35.7 s reading the package)

Performance and validation

Measured 2026-11-08 on the two RTX PRO 6000s with the LIL bench, 30 s cells, c1/2/4 at 0/16K/128K contexts. The package was prepared from local branch glm53-prs (d2b5c5f3 plus preparation PRs #217, #218 and #231; those PRs are unmerged and preparation-only, so serving does not need the branch):

Prefill 8K 16K 64K 128K
tok/s 3,965 4,094 4,160 4,019
Decode, aggregate tok/s C1 C2 C4
0K 185.7 319.4 364.8
16K 184.6 307.3 377.2
128K 194.0 293.5 386.1

Against the same package prepared from upstream 68bbd6dd without the PRs, C2 improved +15.0% / +10.2% / +2.8% at 0K/16K/128K; C1 moved +0.6% / +1.1% / +8.1% and C4 βˆ’2.4% / βˆ’2.6% / +2.9%. The 0K C2 gain occurs with lower acceptance length (2.152 β†’ 1.813, accept rate 0.705 β†’ 0.769), i.e. faster rounds rather than sampling luck. Raw runs are under bench/.

Validation on this package:

  • cached speculative battery: 3 correct + 1 neighbour, 0 corrupt
  • cold 70K Γ—3: 3/3 correct; 350K and ~960K retrieval correct
  • 36 text probes: 0 [gMASK], 0 budget starvation
  • tool calls 6/6; image test correct
  • loads through flare run with the production publisher and 1M context

Reproduction

git clone https://github.com/NVIDIA/hyperflare-inference && cd hyperflare-inference
git checkout d2b5c5f3
git fetch origin pull/217/head:pr-217 pull/218/head:pr-218 pull/231/head:pr-231
git switch -c glm53-prs && git merge --no-edit pr-217 pr-218 pr-231
uv sync --locked --extra prepare --extra test
cargo build --release --manifest-path flare/Cargo.toml
uv pip install --python .venv/bin/python --no-deps ./heliopause

PATH=$PWD/.venv/bin:$PATH .venv/bin/ignition export glm-5.3-flash \
  --gpus 0,1 --max-concurrency 4
.venv/bin/ignition convert CACHE_ENTRY \
  --form packed --output glm53-tp2-unsigned.safetensors

--reuse-weights is optional when a compatible cached default exists. Promote with ignition sign under your own publisher identity before publishing; serving the package needs only flare run, not the preparation environment.

Caveats

  • GB202/SM120, TP2, concurrency 4 and a 4096-token prefill chunk are baked into the package. Re-export for other architectures, TP sizes or capacities.
  • 1,048,576 tokens is the per-request context; the 1,405,952-token pool is shared by up to four concurrent requests, so several full-context conversations cannot be resident at once.
  • The PR package was prepared from unmerged preparation PRs. Every token is verified by the target model, and the correctness battery above is clean; the mainline package remains the conservative reproduction path.
  • The package keeps both cards almost full (~93.8 GiB each, about 98% of usable memory) while loaded. Leave the box otherwise idle, and stop the server with SIGTERM so native replay drains before CUDA teardown.

Credits and provenance

This repository is a packaging, validation and publication effort only. The model, runtime, recipes, kernels, benchmark tooling and the GLM fixes are the work of others. No authorship claim is made over any upstream work.

  • Local Inference Lab and the NVIDIA Hyperflare (formerly Flare) authors and contributors β€” runtime, recipes, package format and documentation (NVIDIA/hyperflare-inference).
  • NVIDIA and Z.ai β€” the GLM-5.3-Flash FP4 checkpoint (nvidia/GLM-5.3-Flash-FP4-Local) and the GLM-5.3-Flash base model.
  • Jason Cook (original-el8) β€” PR #129, the preparation copy-ordering fix that makes GLM speculative decoding correct, and PRs #217/#218/#231, the preparation improvements baked into this package.
  • Local Inference Lab β€” FlashInfer, llm-inference-bench, the b12x mirror (b12x is Luke Alonso's, with Martin Vit and contributors), and the LIL community whose testing and discussion made this bring-up possible.
  • JohannesBertens and JMNI-Labs β€” prior Flare package layouts this repository follows.

Packaged, validated and published on a host with two RTX PRO 6000 GPUs. The package's publisher identity is a signing identity, not an authorship claim. All credit for the model, engine, kernels and fixes belongs to the projects and people listed above.

License

Weights derive from nvidia/GLM-5.3-Flash-FP4-Local and are subject to the NVIDIA Open Model Agreement (license recorded by the exporter: nvidia-open-model-agreement). Review and accept that agreement before use. Hyperflare is Apache-2.0.

Downloads last month
87
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for rickybrent/GLM-5.3-Flash-HYPERFLARE-TP2

Finetuned
(8)
this model