|
|
|
|
|
|
by roadside_picnic
21 days ago
|
|
|
MoEs don't route like most people imagine. They aren't learning topic based experts despite the name The original Mixtral paper [0] (in the "Routing analysis" section) found: "surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic" A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing). 0. https://arxiv.org/pdf/2401.04088 |
|
|