Hacker News new | ask | show | jobs
by skolos 6 days ago
"Multi-token prediction gives a free speedup of up to 2x on many models" - at the expense of halving prompt processing speed
2 comments
hedgehog 6 days ago
MTP has no effect on prompt processing.
link
skolos 6 days ago
Interesting - in my setup (llama.cpp rtx5090 qwen-3.6 27b) prompt processing with mtp is almost half vs non mtp. Sounds like I need to investigate what is wrong.
link
hedgehog 6 days ago
Maybe enabling MTP causes some weights to be displaced to host memory? MTP itself doesn't do anything during prefill so that should be exactly unchanged, decode will vary depending on settings but with 2-4 proposals depending on workload I've never seen an overall slowdown.

edit: I recommend building recent llama.cpp from source, I've been updating about once a week, as there has been a fair amount of work related to MTP recently. If you're running a lot of tool calling on Qwen you might also benefit from one of the bugfixed chat templates like the Froggeric version.

link
tingletech 6 days ago
it works quite will for me in llama.cpp, but I get more like 20% to 40% speedup on tokens per second. I generally use

  spec-type = draft-mtp,ngram-mod
  spec-draft-n-max = 4
I have not observed any effect on prompt processing, which is usually an order of magnitude faster than generation on my spark.
link