[Glemllksdf]

Isn't that some kind of gambling if you offload random experts onto the CPU?

Or is it only layers but that would affect all Experts?

[dragonwriter]

Pretty sure all partial offload systems I’ve seen work by layers, but there might be something else out there.

[namibj]

Speculative decoding is already gambling.