|
|
|
|
|
|
by HarHarVeryFunny
16 days ago
|
|
|
Just to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream). |
|
|