|
|
|
|
|
|
by stymaar
46 days ago
|
|
|
> all you are doing is leaving off a fraction of the most recent assistant message generation True, but not a tiny fraction, qwen is very verbose in its thinking traces. And it basically means that for every (nonthinking) generated token you have to compute the KV twice (once as tg, the second one as pp). |
|
|