[fusionadvocate]

What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human.

If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intelligence, but forget to add steering, or even a chassi.

[dehugger]

I agree. I want to be able to get smaller models which are complete, contained, products which we can run on-prem for our organization.

I'll take a model specialized in web scraping. Give me one trained on generating report and documentation templates (I'd commit felonies for one which could spit out a near-conplete report for SSRS).

Models trained for specific helpdesk tasks ("install a printer", "grant this user access to these services with this permission level").

A model for analyzing network traffic and identifying specific patterns.

None of these things should require titanic models nearing trillions of parameters.

[furyofantares]

This is extremely theorycrafted but I see this as an excellent thing driving AI forward, not holding it back.

I suspect a large part of the reason we've had many decades of exponential improvements in compute is the general purpose nature of computers. It's a narrow set of technologies that are universally applicable and each time they get better/cheaper they find more demand, so we've put an exponentially increasing amount of economical force behind it to match. There needed to be "plenty of room at the bottom" in terms of physics and plenty of room at the top in terms of software eating the world, but if we'd built special purpose hardware for each application I don't think we'd have seen such incredible sustained growth.

I see neural networks and even LLMs as being potentially similar. They're general purpose, a small set of technologies that are broadly applicable and, as long as we can keep making them better/faster/cheaper, they will find more demand, and so benefit from concentrated economic investment.

[fnord123]

They aren't arguing against LLMs They are arguing against their toaster's LLM to make the perfect toast from being trained on the tax policies of the Chang Dynasty.

[int_19h]

Thing is, we keep finding out again and again that having a very broad training mix in the baseline model makes it better across the board, including in those specialized tasks when you fine-tune it.

As I understand it, the general ability to reason is what the models get out of "being trained on the tax policies of the Chang Dynasty", and we haven't really figured out a better way to do so than to throw most everything at them. And even if all you do is make toast, you still need some intelligence.

[fnord123]

> And even if all you do is make toast, you still need some intelligence.

No you don't. That was the point of the example.

[furyofantares]

I'm aware! And I'm personally excited about small models but my intuition is that maybe pouring more and more money into giant general purpose models will have payoff as long as it keeps working at producing better general purpose results (which maybe it won't).

[cruffle_duffle]

That’s just machine learning though!