|
|
|
|
|
|
by randomcatuser
433 days ago
|
|
|
yes, and it's on a per-layer basis, I think! So if the model has 16 transformer layers to go through on a forward pass, and each layer, it gets to pick between 16 different choices, that's like 16^16 possible expert combinations! |
|
|