[furyofantares]

This is extremely theorycrafted but I see this as an excellent thing driving AI forward, not holding it back.

I suspect a large part of the reason we've had many decades of exponential improvements in compute is the general purpose nature of computers. It's a narrow set of technologies that are universally applicable and each time they get better/cheaper they find more demand, so we've put an exponentially increasing amount of economical force behind it to match. There needed to be "plenty of room at the bottom" in terms of physics and plenty of room at the top in terms of software eating the world, but if we'd built special purpose hardware for each application I don't think we'd have seen such incredible sustained growth.

I see neural networks and even LLMs as being potentially similar. They're general purpose, a small set of technologies that are broadly applicable and, as long as we can keep making them better/faster/cheaper, they will find more demand, and so benefit from concentrated economic investment.

[fnord123]

They aren't arguing against LLMs They are arguing against their toaster's LLM to make the perfect toast from being trained on the tax policies of the Chang Dynasty.

[int_19h]

Thing is, we keep finding out again and again that having a very broad training mix in the baseline model makes it better across the board, including in those specialized tasks when you fine-tune it.

As I understand it, the general ability to reason is what the models get out of "being trained on the tax policies of the Chang Dynasty", and we haven't really figured out a better way to do so than to throw most everything at them. And even if all you do is make toast, you still need some intelligence.

[fnord123]

> And even if all you do is make toast, you still need some intelligence.

No you don't. That was the point of the example.

[furyofantares]

I'm aware! And I'm personally excited about small models but my intuition is that maybe pouring more and more money into giant general purpose models will have payoff as long as it keeps working at producing better general purpose results (which maybe it won't).