[3abiton]

This topic has been reposted few times recently, yet never gained much traction. I wonder how much changes there have been between GPT2 to GPT4?

[kridsdale1]

Mixture of Experts model is likely the most significant.

And the scale of everything. GPT3 embedding vectors are around 12,000, vs 768 shown here.

I was curious and the 12k figure closely approximates the median synapse dimensionality of human neurons. Maybe we don’t need much more.