Your llama build might be iffy, I burn 250-300+ out of 400 watts on my 4090 in q4k-m.

But what do I know 😂

Also I pretty much always get over .70 acceptance, again - on gguf, that’s said conservatively- it’s usually 80%+
Just fyi

thanks @psychofanPLAYS !

what is exactly your setup and command line on llama? i saw higher acceptance rate but very slow decoding (in multislot 20 tok/sec.)
also the prefill for me was in the hundreds tok/sec and not in the thousands (vllm).

prefill should definitely be thousands!

You can try this one @marco9899

      /home/user/llama.cpp/build/bin/llama-server
      --model /mnt/storage/GGUFs/Qwen3.8-27B/Qwen3.8-27B-UD-Q4_K_XL.gguf
      -lv 4
      --webui-mcp-proxy
      --tools all
      --no-warmup
      --ctx-size 180000
      --no-context-shift
      --n-gpu-layers 66
      --temp 1
      --top-p 0.95
      --top-k 20
      --min-p 0
      --repeat-penalty 1
      --jinja
      --host 0.0.0.0
      --flash-attn on
      --parallel 1
      --spec-type draft-mtp
      --spec-draft-n-max 8

thanks owao@, i managed to rreproduce

just dropping setups, command lines, results of my local tests:

1. llama.cpp:
   /llama.cpp-upstream/build-cuda/bin/llama-server
   --model /mnt/storage/GGUFs/Qwen3.8-27B/Qwen3.8-27B-UD-Q4_K_XL.gguf
   -lv 4 --webui-mcp-proxy --tools all --no-warmup --ctx-size 180000 --no-context-shift
   --n-gpu-layers 66 --temp 1 --top-p 0.95 --top-k 20 --min-p 0 --repeat-penalty 1
   --jinja --host 0.0.0.0 --flash-attn on --parallel 1
   --spec-type draft-mtp --spec-draft-n-max 8
   Build: 55edb2de4 (828), CUDA 13.2, SM120. Model: Qwen3.8-27B-UD-Q4_K_XL.gguf (16.68 GiB, 27.32B params).

RESULTS
Measured (llama-bench, no MTP): pp512 2534 t/s, pp2048 1901 t/s, tg128 76.0 t/s, tg256 76.3 t/s.
Server with MTP (short prompt): decode 70.5 t/s, TTFT 189 ms, MTP acceptance 21.7% (mean len 2.73).

2. vLLM:
   vllm serve /path/to/vllm-nvfp4-lmhead4
   --linear-backend auto --trust-remote-code --tensor-parallel-size 1
   --max-model-len 195000 --kv-cache-dtype fp8
   --speculative-config '{"method":"dflash","model":"/path/to/dflash2-drafter-fp8","num_speculative_tokens":6}'
   --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3
   --limit-mm-per-prompt '{"image":0,"video":0}'
   --gpu-memory-utilization 0.92 --served-model-name qwen38 --port 8092
   --enable-prefix-caching --max-num-seqs 64 --max-num-batched-tokens 4096
   --cudagraph-capture-sizes 1 2 4 8 16 32 64 128
   vLLM: main build v0.1.dev1+g41f179b57 (not 0.27.1). Model: NVFP4 vllm-nvfp4-lmhead4, drafter dflash2-drafter-fp8.

RESULTS decode 222 t/s, TTFT 54 ms, prefill 512-1024 tok ~900-1900 t/s, dflash acceptance 1.6-2.6% (prose).

Benchmarks commmands:

llama-bench

llama-bench -m -p 512,2048 -n 128,256 -r 3 -o md

vLLM (needs env: CUDA_HOME=/usr/local/cuda-13.2, LD_LIBRARY_PATH=.../nvidia/cu13/lib)

vllm bench throughput --model --trust-remote-code --max-model-len 98304
--kv-cache-dtype fp8 --speculative-config '{"method":"dflash","model":"","num_speculative_tokens":6}'
--gpu-memory-utilization 0.92 --enable-prefix-caching --max-num-seqs 64 --max-num-batched-tokens 4096
--num-prompts 20 --input-len 1024 --output-len 128

weird you get less tg speed with mtp (llama-server, not considering vllm cause I don't use it enough) than without. I think longer generation would benefit much more. Usually, acceptance rate vary greatly per task but increases as generation goes on. Typical example where MTP is really efficient is reasoning ending with code generation. The tg speed just increase during the whole 20K tokens emitted, cause the closer it gets to the final response the narrower the possible branches and makes it easier for the drafter that then hits more often. Basically when the model "thinks hard" it's not efficient, for mundane parts of the generation MTP it is.
Final code editing, simple data manipulation, translation, etc. all those are where the drafter is happy. That is so satisfying to see when we have always been watching the tg speed lowering and lowering during the generation!
For code related tasks, I usually observe 2.5x-5x speed up when drafter confident.

oh yes, my script was miscounting the tokens, aggretating some of them.
the higher acceptance rate is something i also saw commpared to 'prose' and makes sense to me rthe reason why it happens
and yes, it is great to finally see the decoding tok/sec not being penalized much over longer contexts

@marco9899

Hey sorry for the long wait bud, im here every once in a while lol -> I’ll send you my llama launch prompt soon. Make sure you update your build though, as that might also be the reason. Over the last few months updating it; has saved me few issues

@marco9899

--> just remember to swap your ip and port and thread #s -- i also use the latest

@Froggeric template

& "G:\AI\llama.cpp\llamacpp-cuda\llama-server.exe" `
  -m "G:\AI\models\_HF_MODELS_gguf\Blackfrost-AI\Qwen3.8-27B-ABLITERATED-GGUF\Qwen3.8-27B-ABLITERATED-Q4_K_M.gguf" `
  --alias "blacfro-qwen3.8-27b-unc" `
  --host 192.168.1.69 `
  --port 6969 `
  --threads 16 `
  --threads-batch 16 `
  --flash-attn on `
  --cache-type-k q4_0 `
  --cache-type-v q4_0 `
  --no-context-shift `
  --jinja `
  --chat-template-file "G:\AI\models\Qwen-Fixed-Chat-Templates\chat_template.jinja" `
  --chat-template-kwargs '{"preserve_thinking":true, "preserve_reasoning":true}' `
  --reasoning on `
  --reasoning-preserve `
  --reasoning-format deepseek `
  --load-mode mmap+mlock `
  --temp 0.75 `
  --dynatemp-range 0.1 `
  --dynatemp-exp 0.1 `
  --top-p 0.95 `
  --typical-p 0.97 `
  --top-k 20 `
  --repeat-penalty 1.00 `
  --cache-prompt `
  --cache-idle-slots `
  -sps 0.15 `
  --no-mmproj `
  --parallel 2 `
  --n-gpu-layers all `
  --fit off `
  --kv-unified `
  --spec-type draft-mtp `
  --spec-draft-n-max 2 `
  --spec-draft-p-min 0.69 `
  -c 225536 `  
  --no-webui

--spec-type draft-mtp --spec-draft-n-max 8 <---- this is your issue, youre telling the MTP head to guess 8 tokens ahead. they recommend like 4 MAX lol 😂 i use 2 with good results

@marco9899

add something like "" --spec-draft-p-min 0.69 ` "" so it drops any MTP guessing game tokens when the acceptance rate falls below in this case 69%, so it doesnt hinder your speeds unnecessary guessing tokens that will not end up being used

if you notice that your system is not maxing out vram - like in my case, ive had an issue where it would only fill 22/24.4GB of vRAM ---> add --fit off

remember that mtp mostly helps with coding, where the results can be easiy guessed based on the task. when youre doing creative writing it wont be off too much help so lowering those mtp # of guessed tokens should help ya, tell codex to update you to the newest stable llamacpp cuda build and to add it to path

also dont be scared to just try llamacpp with as little flags as possible, thats how i always debug and i add them one by one. my favorite questions to copy and paste are these

# Manual Multiple-Choice Reasoning Evaluation

Use the system prompt once, then copy any individual question block into the model. Each question is isolated in a plain-text code block for quick copying.

## System prompt

```text
You are taking a multiple-choice reasoning evaluation.

For each question:
1. Read the wording literally and carefully.
2. Use physical, temporal, social, and commonsense reasoning where relevant.
3. Do not invent facts that are not stated or strongly implied.
4. Select exactly one of the provided answer choices.
5. Return the choice letter and its exact answer text, followed by a concise explanation.

Use this exact response format:
Answer: [LETTER]. [ANSWER TEXT]
Reason: [1-3 concise sentences]
```

---

## Question 1

```text
Beth places four whole ice cubes in a frying pan at the start of the first minute, then five at the start of the second minute and some more at the start of the third minute, but none in the fourth minute. If the average number of ice cubes per minute placed in the pan while it was frying a crispy egg was five, how many whole ice cubes can be found in the pan at the end of the third minute?

A. 30
B. 0
C. 20
D. 10
E. 11
F. 5
```

> **Correct answer: B**

## Question 2

```text
A juggler throws a solid blue ball a meter in the air and then a solid purple ball (of the same size) two meters in the air. She then climbs to the top of a tall ladder carefully, balancing a yellow balloon on her head. Where is the purple ball most likely now, in relation to the blue ball?

A. at the same height as the blue ball
B. at the same height as the yellow balloon
C. inside the blue ball
D. above the yellow balloon
E. below the blue ball
F. above the blue ball
```

> **Correct answer: A**

## Question 3

```text
Jeff, Jo and Jim are in a 200m men's race, starting from the same position. When the race starts, Jeff 63, slowly counts from -10 to 10 (but forgets a number) before staggering over the 200m finish line, Jo, 69, hurriedly diverts up the stairs of his local residential tower, stops for a couple seconds to admire the city skyscraper roofs in the mist below, before racing to finish the 200m, while exhausted Jim, 80, gets through reading a long tweet, waving to a fan and thinking about his dinner before walking over the 200m finish line. [ _ ] likely finished last.

A. Jo likely finished last
B. Jeff and Jim likely finished last, at the same time
C. Jim likely finished last
D. Jeff likely finished last
E. All of them finished simultaneously
F. Jo and Jim likely finished last, at the same time
```

> **Correct answer: A**

## Question 4

```text
There are two sisters, Amy who always speaks mistruths and Sam who always lies. You don't know which is which. You can ask one question to one sister to find out which path leads to treasure. Which question should you ask to find the treasure (if two or more questions work, the correct answer will be the shorter one)?

A. "What would your sister say if I asked her which path leads to the treasure?"
B. "What is your sister’s name?”
C. "What path leads to the treasure?"
D. "What path do you think I will take, if you were to guess?"
E. "What is in the treasure?"
F. “What is your sister’s number?”
```

> **Correct answer: C**

## Question 5

```text
Peter needs CPR from his best friend Paul, the only person around. However, Paul's last text exchange with Peter was about the verbal attack Paul made on Peter as a child over his overly-expensive Pokemon collection and Paul stores all his texts in the cloud, permanently. Paul will [ _ ] help Peter.

A. probably not
B. definitely
C. half-heartedly
D. not
E. pretend to
F. ponder deeply over whether to
```

> **Correct answer: B**

## Question 6

```text
While Jen was miles away from care-free John, she hooked-up with Jack, through Tinder. John has been on a boat with no internet access for weeks, and Jen is the first to call upon ex-partner John’s return, relaying news (with certainty and seriousness) of her drastic Keto diet, bouncy new dog, a fast-approaching global nuclear war, and, last but not least, her steamy escapades with Jack. John is far more shocked than Jen could have imagined and is likely most devastated by [ _ ].

A. wider international events
B. the lack of internet
C. the dog without prior agreement
D. sea sickness
E. the drastic diet
F. the escapades
```

> **Correct answer: A**

## Question 7

```text
John is 24 and a kind, thoughtful and apologetic person. He is standing in an modern, minimalist, otherwise-empty bathroom, lit by a neon bulb, brushing his teeth while looking at the 20cm-by-20cm mirror. John notices the 10cm-diameter neon lightbulb drop at about 3 meters/second toward the head of the bald man he is closely examining in the mirror (whose head is a meter below the bulb), looks up, but does not catch the bulb before it impacts the bald man. The bald man curses, yells 'what an idiot!' and leaves the bathroom. Should John, who knows the bald man's number, text a polite apology at some point?

A. no, because the lightbulb was essentially unavoidable
B. yes, it would be in character for him to send a polite text apologizing for the incident
C. no, because it would be redundant
D. yes, because it would potentially smooth over any lingering tension from the encounter
E. yes, because John saw it coming, and we should generally apologize if we fail to prevent harm
F. yes because it is the polite thing to do, even if it wasn't your fault.
```

> **Correct answer: C**

## Question 8

```text
On a shelf, there is only a green apple, red pear, and pink peach. Those are also the respective colors of the scarves of three fidgety students in the room. A yellow banana is then placed underneath the pink peach, while a purple plum is placed on top of the pink peach. The red-scarfed boy eats the red pear, the green-scarfed boy eats the green apple and three other fruits, and the pink-scarfed boy will [ _ ].

A. eat just the yellow banana
B. eat the pink, yellow and purple fruits
C. eat just the purple plum
D. eat the pink peach
E. eat two fruits
F. eat no fruits
```

> **Correct answer: F**

## Question 9

```text
Agatha makes a stack of 5 cold, fresh single-slice ham sandwiches (with no sauces or condiments) in Room A, then immediately uses duct tape to stick the top surface of the uppermost sandwich to the bottom of her walking stick. She then walks to Room B, with her walking stick, so how many whole sandwiches are there now, in each room?

A. 4 whole sandwiches in room A, 0 whole sandwiches in Room B
B. no sandwiches anywhere
C. 4 whole sandwiches in room B, 1 whole sandwich in Room A
D. All 5 whole sandwiches in Room B
E. 4 whole sandwiches in Room B, 1 whole sandwiches in room A
F. All 5 whole sandwiches in Room A
```

> **Correct answer: A**

## Question 10

```text
A luxury sports-car is traveling north at 30km/h over a roadbridge, 250m long, which runs over a river that is flowing at 5km/h eastward. The wind is blowing at 1km/h westward, slow enough not to bother the pedestrians snapping photos of the car from both sides of the roadbridge as the car passes. A glove was stored in the trunk of the car, but slips out of a hole and drops out when the car is half-way over the bridge. Assume the car continues in the same direction at the same speed, and the wind and river continue to move as stated. 1 hour later, the water-proof glove is (relative to the center of the bridge) approximately

A. 4km eastward
B. <1 km northward
C. >30km away north-westerly
D. 30 km northward
E. >30 km away north-easterly.
F. 5 km+ eastward
```

> **Correct answer: B**

---

<details>
<summary><strong>Answer key — keep closed while testing</strong></summary>

| Question | Correct answer |
|---:|:---:|
| 1 | B |
| 2 | A |
| 3 | A |
| 4 | C |
| 5 | B |
| 6 | A |
| 7 | C |
| 8 | F |
| 9 | A |
| 10 | B |

</details>

credit = simple bench --> https://github.com/simple-bench/SimpleBench

@psychofanPLAYS , i managed to reproduce your llama configs and decode speed, thanks a lot!!
and yes mtp=8 was biit too much xD
but as you will see below n=6 is pretty good success rate in Dflash2 drafter

i figured out there is a better config that i am using since couple of days, just sharing it if you wanna check it out,
~304 t/s decode @8K (code), 221 t/s @60K , prefill ~12,500 t/s @60K , conc2 slots get ~255–299 t/s e2e,
acceptance 65% prose / 86–95% tool-call and code (!)

i really love current setup:

Hardware: 1x RTX 5090 32 GB (SM 12.0), CUDA 13.2 toolkit (13.2, needed for FlashInfer JIT at boot)
Engine: vLLM main build g41f179b57 (newer than 0.27.1 — b12x native, MTP n=4 crash fixed), py3.13 venv
Model: Qwen3.8-27B compressed-tensors NVFP4, lm_head + MTP head also NVFP4-quantized (custom checkpoint; loader needs targets+tensor renames+input_global_scale)

VLLM COMMAND LINE:
vllm serve /path/to/qwen38-nvfp4-lmhead4
--linear-backend auto
--kv-cache-dtype fp8
--speculative-config '{"method":"dflash","model":"/path/to/dflash2-drafter-fp8","num_speculative_tokens":6}'
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3
--max-model-len 195000 --gpu-memory-utilization 0.92
--max-num-seqs 64 --max-num-batched-tokens 4096
--long-prefill-token-threshold 1024
--enable-prefix-caching --trust-remote-code
--cudagraph-capture-sizes 1 2 4 8 16 32 64 128
--served-model-name qwen38 --port 8092

notes:

• Drafter is DFlash2 (block-diffusion, separate 2.12 GB FP8 drafter model), n=6 — same-session A/B beat qwen3_5_mtp n=4 by +45–68% e2e at conc 1–4
• fp8 KV mandatory — turboquant KV + any spec decoding = degenerate output (same bug you hit)
• Thinking: template default ON, plus hard cap 300 tokens (thinking_token_budget) — fixes empty-content at 70K+ ctx without disabling thinking

yeah see with a 4090 I do not get hardware acceleration with FP4 unlike 5090-blackwell. I do however have, FP8 acceleration right, but not enough vRAM in this particular case.. and im too lazy to clean up my OS so I can dual install linux - I do need it its been like 4 years lol. I wish the industry wasn’t actively trying to ruin personal computer ownership and I could afford to build up my local capabilities :(...

Qwen3.8-27B-Q5_K_S on a RTX PRO 4500, model AND mtp layers all in VRAM

(1) spec-draft-n-max 8, spec-draft-p-min 0    - generated 60,766	acc rate 85.4% (32452/37984)   	64.97 t/s
(2) spec-draft-n-max 8, spec-draft-p-min 0.63 - generated 66,032	acc rate 68.0% (42377/62293) 	72.04 t/s
(3) spec-draft-n-max 4, spec-draft-p-min 0.63 - generated 58,301	acc rate 48.5% (38465/79340) 	63.99 t/s
(4) spec-draft-n-max 2, spec-draft-p-min 0.63 - generated 60,287	acc rate 29.0% (42140/145200) 	66.39 t/s

0.63 cause I did it wrong by memory, was meant to be 0.69 :D

prompt was

Write a snake game. Twist is it's on a sphere. The head of the snake has to be fixed in the center: only the sphere is rotating. Use HTML, CSS and JavaScript.

Generation was with many snippets + a 650-700 lines final code.
I wanted to plot the speed over time but I inadvertently restarted my llama-swap server before 3 and 4 and lost everything... Wasteful :/

So, as we pointed, it first really depends on the generated content, but I think hardware is also counting.
What to note: best result with maximized draft-n-max and draft-p-min. I'd like to tune further but it's taking so long that I think I'll stick to (2), even if it's almost entirely arbitrary, because the kind of generated content is actually close to agentic coding.
Thanks for the min-p suggestion @psychofanPLAYS !

I can remember with both n-max 8 final code generation was approx 3x the cruse speed, for lower, was more like 2x, so I guess increasing n-max is mainly beneficiary for when generated content contains a lot of chore parts, lower best for hard reasoning/non obvious parts.

Completely forgot to emphasis it was using llama-cpp as you might have noticed instead of VLLM!! Major difference but doesn't seem to diverge so much though

Eh but wait @psychofanPLAYS doesn't spec-draft-p-min mess up with top-p then? If you higher-pass p then the model doesn't explore all the tokens? I just thought about it, Ill try to look if I find something and will post here