iAcloud/gemma4-31B-it-rbln-tp16

gemma-4-31B-it — RBLN (ATOM CA25) compiled

google/gemma-4-31B-it (multimodal, image-text-to-text) pre-compiled for Rebellions RBLN-CA25 NPUs with optimum-rbln, ready to serve with vllm-rbln. This repo contains the compiled artifact only — you need RBLN NPUs and the RBLN SDK to run it.

Hardware & software

NPU	16× RBLN-CA25 (language model: 16 devices, vision tower: 8 devices)
Compiler	rebel-compiler==0.11.0
Runtime	optimum-rbln==0.11.0.post1, vllm-rbln==0.11.0, transformers==5.8.1

Compile configuration

Option	Value
max_seq_len	131,072
batch_size	8
decoder_batch_sizes (dynamic batch)	[1, 2, 4, 8]
attn_impl / kvcache_partition_len	flash_attn / 16384
cache_impl / sliding_window	hybrid / 1024
kvcache_num_blocks	32 (≈524k shared KV tokens → ~4 concurrent full-131k sequences)
vision max_soft_tokens / image_prefill_chunk_size	280 / 384 (tuned for ≤3 MB images)

kvcache_num_blocks is pinned to 32 (not auto): auto fills every device to ~15.7 GiB, leaving no room for the vision tower on the shared devices at serve time. 32 leaves ~1.3 GiB/device of headroom.

Serving

vllm serve <this-repo-or-local-path> \
  --max-num-seqs 8 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

Notes:

• --max-num-seqs 8 is required — vllm-rbln defaults it to 1.
• Do not pass --tensor-parallel-size; the 16-device split is baked into the compiled model (vllm-rbln forces TP=1).
• Reasoning (thinking) is per-request: send chat_template_kwargs={"enable_thinking": true}; reasoning text is returned in message.reasoning_content.

License

Derived from google/gemma-4-31B-it; use is subject to the Gemma Terms of Use.