[ashleyn]

Is there active research in improving the density of models i.e. higher performance at lower parameter counts? Is this something that's even theoretically possible to do with the architecture?

I feel like I can have a much better model for how I can expect the cost of LLMs to change over the next few years. "Build more datacenters" reeks of "just one more lane bro" and because of Jevons paradox I'm skeptical it will bring costs down. Could ASICs and custom processors (ala Groq) move the needle here at all if the model density can't be improved?

[skybrian]

Taalas has a pretty impressive proof of concept, but there's been no news about them recently.

https://chatjimmy.ai/

[wmf]

Yes, LLM density has continuously improved. A recent 32B model beats older >100B models. It may level off at some point but no one knows when that will be.