[Grombobulous]

Moore's law is over in the literal sense but silicon continues to advance relatively quickly.

This claim isn't really outlandish in any way. It's not hard to imagine:

- Future models being able to handle current frontier models' workflows with much higher efficiency.

- Future consumer devices like phones having 2-4x the RAM onboard along with GPU/NPU performance greatly increased in 10-15 years.

[dmoy]

Moore's law for 10-15 years was more like 20-100x ram sizes, not 2-4x.

Performance, storage, etc is definitely getting better, but it's a different scale of improvement

[msdz]

10 to 15 years from now the scale of LLM efficiency improvements is, quite literally, unpredictable.

It could be that the company valuations crash tomorrow, and (almost) only performance gains achievable on hobbyist-level hardware come to fruition from there on out.

Or it could be that in the future, we have a custom "model FPGA" à la Taalas [0] in every home, and that it turns out we can still massively boost inference efficiency due to novel discoveries like TurboQuant [1] or a somehow-improved quantization method [2] again and again ten times over.

Point is, Moore's law in this context shouldn't be applied to just hardware spec sheets alone, but more the total number of "parameters potentially improving", IMO.

[0] https://chatjimmy.ai

[1] https://research.google/blog/turboquant-redefining-ai-effici...

[2] https://prismml.com/news/bonsai-27b