anemll/Hy4-preview-FlashMoE-STQ1_0

Hy4-preview Flash-MoE STQ1_0

SSD-backed dense + expert-sidecar package for Tencent's Hy4-preview, prepared from AngelSlim/Hy4-preview-GGUF.

This is not a standalone GGUF. Non-routed and shared tensors are stored in model-dense.gguf; the 77 MoE layers' routed gate, up, and down tensors are stored in the expert-major sidecar/ directory and streamed from SSD into a slot bank on demand.

Use the HY4-1.25-bit branch of anemll-flash-llama.cpp. Stock llama.cpp does not understand the hyv4 runtime graph, STQ1_0 type 43, or this dense + sidecar layout.

Package details

Property	Value
Architecture	hyv4
Transformer blocks	78
Leading dense blocks	1
Routed MoE blocks	77
Routed experts per MoE block	256
Native selected experts per token	8
Shared experts per MoE block	1
Embedding width	6144
MoE intermediate width	2048
Dense/shared tensors	1,903
Dense/shared GGUF size	22,645,374,580 bytes
Routed tensors	231
Routed sidecar size	206,762,409,984 bytes

The routed sidecar preserves the source GGUF payload bytes exactly. No routed expert was dequantized or requantized:

• 29 layers: STQ1_0 gate/up and IQ3_XXS down.
• 45 layers: IQ2_XXS gate/up and IQ3_XXS down.
• 3 layers: IQ2_XXS gate/up and IQ4_XS down.

STQ1_0 is GGML type 43. Its stored block is 42 bytes per 256 weights, or 1.3125 bits per weight including the block scale. The complete mixed-format source model averages approximately 2.38 bits per weight.

All 231 sidecar entries were compared byte-for-byte with the source GGUF after packaging.

How the STQ1_0 packing works

Each STQ1_0 block covers 256 scalar weights, arranged as 64 four-weight groups. Every group has exactly one zero and three values in {-d, +d}, so there are 4 x 2^3 = 32 possible patterns. A pattern therefore needs five bits. The on-disk ABI stores those five bits as a four-bit codebook index plus a one-bit table selector; the bit fields are not stored literally as separate zero-position and sign fields.

The 64 patterns consume 40 bytes and the entire 256-weight block shares one FP16 (binary16) scale d, which consumes another two bytes:

64 groups x 5 bits = 320 bits = 40 bytes
one shared FP16 scale          =  2 bytes
total                            42 bytes / 256 weights = 1.3125 bpw

The graphic's PTQ label refers specifically to AngelSlim's conversion of the released BF16 Hy4 checkpoint into this MIX-STQ1_0 GGUF. The original Sherry paper presents its 3:4 ternary method in a quantization-aware-training setting; it does not define Hy4's 29/48 layer split. For this artifact, AngelSlim describes the layer choice as importance-matrix-derived: 29 expert gate/up layer pairs use STQ1_0 and the remaining 48 use ordinary IQ2_XXS PTQ at 2.0625 bpw. Calling the latter "sensitive" is an interpretation of that selection, not wording from the paper. Down projections are IQ3_XXS except for the last three layers, which use IQ4_XS.

Download

The complete repository is approximately 214 GiB. Download it to a fast, directly attached SSD while preserving the directory layout:

hf download anemll/Hy4-preview-FlashMoE-STQ1_0 \
  --local-dir ~/Models/HY4/Hy4-preview-Flash-STQ1_0

Expected layout:

~/Models/HY4/Hy4-preview-Flash-STQ1_0/
├── LICENSE
├── NOTICE
├── README.md
├── model-dense.gguf
├── flashmoe-package.json
└── sidecar/
    ├── manifest.json
    ├── layer_001.bin
    ├── ...
    └── layer_077.bin

Network volumes and slow external links will substantially reduce decode performance because slot misses issue direct sidecar reads.

Build for Apple Silicon

git clone https://github.com/Anemll/anemll-flash-llama.cpp
cd anemll-flash-llama.cpp
git checkout HY4-1.25-bit

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_METAL=ON \
  -DLLAMA_FLASH_MOE_GPU_BANK=ON \
  -DLLAMA_BUILD_TESTS=ON \
  -DBUILD_TESTING=ON

cmake --build build \
  --target llama-cli test-flashmoe-slot8-hyv4 \
  -j 8

./build/bin/test-flashmoe-slot8-hyv4

The HY4 Metal numerical test covers all four supported gate/up and down type combinations at both small and real HY4 shapes. The validated build reports 10/10 cases passed.

M5 Max, 128 GB: 96-slot run

One cached expert slot across all 77 MoE layers occupies 807,665,664 bytes (770.25 MiB). A 96-slot bank therefore reserves approximately 72.21 GiB for routed experts. Together with the 21.09 GiB dense/shared GGUF, this configuration uses about 93.3 GiB before OS and runtime overhead. It is intended for a 128 GB M5 Max with adequate free memory.

LLAMA_FLASH_MOE_EXPERIMENTAL_CPU_VISIBLE_SLOT_WRITES=1 \
LLAMA_FLASH_MOE_EXPERIMENTAL_PARALLEL_SLOT_READS=1 \
./build/bin/llama-cli \
  -m ~/Models/HY4/Hy4-preview-Flash-STQ1_0/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/HY4/Hy4-preview-Flash-STQ1_0/sidecar \
  --moe-slot-bank 96 --moe-topk 8 --moe-cache-io-split 4 --slot8 \
  -fit on -ub 1 -b 1 -c 2048 -ngl 999 \
  --no-warmup -st --temp 0 --seed 1 \
  -p "Make a game of Tetris in HTML" -n 256 --perf

The larger bank caches more experts and can reduce repeated SSD reads. Close other memory-heavy applications before running. If the fitter cannot leave enough headroom, use the 8-slot configuration below.

On an M5 Max 128 GB, the command above measured 3.2 prompt tokens/s and 4.8 generation tokens/s over 255 decoded tokens. Keep macOS Low Power Mode off and use adequate AC power for reproducible sustained performance. The summary should report cpuvis=on preads=on batchrd=on, zero expert-upload time, and a nonzero fused count with reference=0.

Memory-saving: 8-slot run

Eight slots are the minimum for the model's native top-8 route and reserve approximately 6.02 GiB for routed experts. Dense/shared weights plus the slot bank occupy about 27.1 GiB before OS and runtime overhead. This mode minimizes memory use but causes substantially more SSD traffic and slot eviction.

LLAMA_FLASH_MOE_EXPERIMENTAL_CPU_VISIBLE_SLOT_WRITES=1 \
LLAMA_FLASH_MOE_EXPERIMENTAL_PARALLEL_SLOT_READS=1 \
./build/bin/llama-cli \
  -m ~/Models/HY4/Hy4-preview-Flash-STQ1_0/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/HY4/Hy4-preview-Flash-STQ1_0/sidecar \
  --moe-slot-bank 8 --moe-topk 8 --moe-cache-io-split 4 --slot8 \
  -fit on -ub 1 -b 1 -c 128 -ngl 999 \
  --no-warmup -st --temp 0 --seed 1 \
  -p "Hello" -n 16 --perf

The 8-slot configuration was validated on an M5 Max and reported Flash-MoE routed src=pread-slot-bank. The initial cold smoke measured about 1.4 prompt tokens/s and 1.7 generation tokens/s; this is a functional baseline, not a guaranteed benchmark.

Important runtime choices:

• Top-K 8 is native model metadata. Keep --moe-topk 8; more cache slots do not change routing width.
• -b 1 -ub 1 selects the conservative decode-oriented SSD path.
• --moe-cache-io-split 4 splits expert reads into bounded chunks.
• --slot8 selects the validated fused Metal top-8 operator. At shutdown, require flashmoe_slot8 fused=N ... reference=0 with N > 0.
• The two environment variables enable direct pread() into the shared Metal slot bank and parallel reads for independent misses. They improved the checked 96-slot run from approximately 3.8-3.9 to 4.7 generation tokens/s; the longer charged run above reached 4.8 tokens/s.
• The current temporal-prefetch heuristic did not improve the checked HY4 trace and is intentionally omitted.
• Do not use --slot4. It changes the model's native routed width.

Current runtime boundaries

• The branch supports ordinary autoregressive generation but not HY4's native MTP/speculative layer.
• HY4 DSA/indexer weights remain in the dense GGUF, but the first runtime port uses full MLA attention. It is equivalent while visible causal history is at most the model's 2048-key indexer window; beyond that it is a full-attention fallback, not the advertised sparse 1M-context implementation.
• Keep initial validation contexts at or below 2048 tokens.

Modification notice, attribution, and license

Hy4-preview was developed by the Tencent Hy Team.

• Original model: tencent/Hy4-preview
• Source GGUF: AngelSlim/Hy4-preview-GGUF
• Flash-MoE packaging/runtime: Anemll/anemll-flash-llama.cpp, HY4-1.25-bit

Changes made for this distribution: the source GGUF was split into a dense/shared GGUF and an expert-major sidecar. The 231 routed tensors were removed from model-dense.gguf and their original quantized bytes were copied into 77 layer sidecars. Package and tensor manifests were added. The routed weights themselves were not dequantized, requantized, or otherwise altered.

Copyright (C) 2026 Tencent. All rights reserved.

Tencent Hy4-preview is licensed under the Apache License 2.0. This repository includes the upstream license verbatim in LICENSE. Redistribution and use must comply with that license. The names and trademarks of Tencent and other parties are used only to identify the origin of the model; this package is not an endorsement by Tencent.

Verification metadata

flashmoe-package.json records dense-package and runtime hints. sidecar/manifest.json records source tensor names, offsets, quant types, shapes, exact byte lengths, expert counts, and expert-major offsets.