[ycui1986]

There are a lot of SSD streaming engines these days. But few to actually try some hard features.

There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.

If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.

[zozbot234]

> The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.

When using SSD streaming, the GPU is practically always waiting for the SSD to fetch the right expert, rather than the other way around. There is basically zero slack on the SSD side, so I'm not sure how "prefetching" is supposed to help. It would mostly hurt by fetching the wrong predicted experts, which already makes conventional MTP practically unhelpful for typical (not widely batched) SSD streamed inference.

[hacklas]

Worth mentioning why this is harder than it looks.

There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.

The router needs to look at the state produced by the experts below it.

Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.

So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.

[IsTom]

That kind of sounds like a branch predictor in a CPU.

[ycui1986]

exactly. we need branch predictor for expert weight.