[AlexCoventry]

MoE models route each token, in every transformer layer, to a set of specialized feed-forward networks (fully-connected perceptrons, basically), based on a score derived from the token's current representation.

[neom]

Good visual explainer in here: https://deepgram.com/learn/mixture-of-experts-ml-model-guide