|
|
|
|
|
|
by ac29
50 days ago
|
|
|
Modern GPUs aren't optimized for MoEs though? The advantage to a dense model like this Mistral one is that it is as smart as a much larger MoE model so it can fit on less GPUs. The tradeoff is that it is much slower since it has to read 100% of its weights for every token, MoE models typically only read about a tenth (though sparsity levels vary). |
|
|