dahara1/gemma-4-E4B-it-assistant_gguf

Gemma4は小さいモデルに先行させる事で推論を高速化するMTP(Multi-Token Prediction)という仕組みがあります。
このモデルはdahara1/gemma-4-E4B-it-UD-japanese-imatrixをMTPで動かすための小モデルです。

Gemma4 has a mechanism called MTP (Multi-Token Prediction) that speeds up inference by running a smaller model before the main model.
This model is a small model for running dahara1/gemma-4-E4B-it-UD-japanese-imatrix using MTP.

llama-server command example

E4Bでは現在、明示的に--flash-attn offにする必要があります。
詳細はissues/24400をご覧ください

In E4B, you now need to explicitly set --flash-attn off.
See issues/24400 for details.

../llama.cpp/build/bin/llama-server   \
-m ./gemma-4-E4B-it-UD-Q4_K_XL.gguf   \
--host 0.0.0.0   \
--port 8080   \
--temp 1.0   \
--top-p 0.95   \
--top-k 64   \
--min-p 0.0  \
--ctx-size 21000   \
--flash-attn off   \
--jinja  \
--chat-template-file chat_template.jinja   \
--model-draft ./gemma-4-E4B-it-assistant/Gemma-4-E4B-It-Assistant-78M-Q8_0.gguf  \
--spec-type draft-mtp \
--reasoning on

ベンチマーク結果 RTX 4060ti(16GB)

ドラフトなしモデル
mtp_bench.py
  code_python        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.4
  code_cpp           pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.7
  explain_concept    pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.7
  summarize          pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.5
  qa_factual         pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.4
  translation        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.4
  creative_short     pred=  35 draft=   0 acc=   0 rate=n/a tok/s=70.4
  stepwise_math      pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.2
  long_code_review   pred= 192 draft=   0 acc=   0 rate=n/a tok/s=67.2

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1571,
  "total_draft": 0,
  "total_draft_accepted": 0,
  "aggregate_accept_rate": null,
  "wall_s_total": 24.68
}

mtp_bench_jp.py
  code_python        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=69.3
  code_cpp           pred= 192 draft=   0 acc=   0 rate=n/a tok/s=68.0
  explain_concept    pred= 192 draft=   0 acc=   0 rate=n/a tok/s=65.6
  summarize          pred=  47 draft=   0 acc=   0 rate=n/a tok/s=65.3
  qa_factual         pred= 192 draft=   0 acc=   0 rate=n/a tok/s=67.7
  translation        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=66.9
  creative_short     pred= 192 draft=   0 acc=   0 rate=n/a tok/s=68.0
  stepwise_math      pred= 192 draft=   0 acc=   0 rate=n/a tok/s=68.6
  long_code_review   pred= 192 draft=   0 acc=   0 rate=n/a tok/s=65.7

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1583,
  "total_draft": 0,
  "total_draft_accepted": 0,
  "aggregate_accept_rate": null,
  "wall_s_total": 25.43
}

ドラフトありモデル
  code_python        pred= 192 draft= 183 acc= 130 rate=0.710 tok/s=157.5
  code_cpp           pred= 192 draft= 185 acc= 128 rate=0.692 tok/s=153.8
  explain_concept    pred= 192 draft= 243 acc= 110 rate=0.453 tok/s=121.2
  summarize          pred= 192 draft= 196 acc= 125 rate=0.638 tok/s=146.6
  qa_factual         pred= 192 draft= 193 acc= 126 rate=0.653 tok/s=148.7
  translation        pred= 192 draft= 226 acc= 115 rate=0.509 tok/s=128.1
  creative_short     pred=  38 draft=  72 acc=  14 rate=0.194 tok/s=81.5
  stepwise_math      pred= 192 draft= 192 acc= 126 rate=0.656 tok/s=147.5
  long_code_review   pred= 192 draft= 250 acc= 107 rate=0.428 tok/s=113.9

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1574,
  "total_draft": 1740,
  "total_draft_accepted": 981,
  "aggregate_accept_rate": 0.5638,
  "wall_s_total": 13.48
}
  code_python        pred= 192 draft= 201 acc= 124 rate=0.617 tok/s=144.8
  code_cpp           pred= 192 draft= 166 acc= 134 rate=0.807 tok/s=169.0
  explain_concept    pred= 192 draft= 232 acc= 113 rate=0.487 tok/s=125.7
  summarize          pred=  51 draft=  54 acc=  33 rate=0.611 tok/s=142.1
  qa_factual         pred= 192 draft= 191 acc= 127 rate=0.665 tok/s=150.6
  translation        pred= 192 draft= 225 acc= 115 rate=0.511 tok/s=129.2
  creative_short     pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=121.1
  stepwise_math      pred= 192 draft= 204 acc= 123 rate=0.603 tok/s=142.3
  long_code_review   pred= 192 draft= 221 acc= 117 rate=0.529 tok/s=127.7

Aggregate: {
  "n_requests": 9,
  "total_predicted": 1587,
  "total_draft": 1734,
  "total_draft_accepted": 996,
  "aggregate_accept_rate": 0.5744,
  "wall_s_total": 13.7
}