[gortok]

> The only thing that matters is if LLMs with sufficient scaling can become frontier AI researchers kicking off the exponential. Everything else is transient

As long as the term “AI” means by-and-large LLMs with additional features sprinkled on top, the answer is no. More likely (without careful vetting by the folks aggregating these models) is that the quality will go down as more and more AI-generated output gets subsumed into these models.

Even without that particular problem, LLMs-as-AI can only give us probabilistic outputs based on inputs; and by definition they’re reliant on humans to provide the training data for their model. Without specialized knowledge or training on that knowledge (And even with it, viz. Meta’s engineering), we don’t have to worry about AI itself. We do have to worry what investors who are looking for outsized returns will do to get those returns, job market be damned.

The problem for us isn’t that AI will take our jobs; it’s that snake-oil salesmen can sell the idea that AI will take our jobs, investors buy into it, companies try it, fire their folks, the snake-oil salesmen IPOs, the companies that bought into this idea implode in some form or fashion, and the salesmen have already taken the money and ran. Of course, we still lose our jobs, but maybe (!) we get them back when this all fails?

[openquery]

> More likely (without careful vetting by the folks aggregating these models) is that the quality will go down as more and more AI-generated output gets subsumed into these models.

This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.

How much do these models need to do before people throw their hands in the air and say, ok this is happening. The Erdos unit distance problem, which as far as I understand was approached by multiple competent mathematicians was solved by a frontier model. Sure people argue there was no novelty there (I cannot comment as a non-mathematician) but it feels like they can draw lines laterally from deep knowledge in different fields (in this case combinatorics and algebraic number theory I believe) and solve problems.

Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).

[taurath]

> How much do these models need to do before people throw their hands in the air and say, ok this is happening

What is "this"? Most people arguing against some of the more fervent predictions and promises of "inevitability" are people who are using these models in day to day - they see what the models can do, and what they struggle at.

> Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).

My genuine prediction is that you'll get a lot of early results simply because you're applying attention to some low hanging fruit of problems, but then it will drop off due to the cost of tokens and the low rate of return. This doesn't mean that the models are especially capable of novel thought, just that we haven't algorithmically brute forced a problem with known solutions.

We would be seeing more success cases if the promises were true, setting aside AGI, human replacement, etc. We would see more, better products with more features that people would use. We wouldn't be having any arguments. The human replacement presupposes the models work in ways that they don't, and until proven otherwise, can't. I've watched those who embrace it fully flounder around on projects, some have lost their mind from the constant LLM validation, and I've seen companies go all in and then pull back based on both cost and efficacy over the last year.

I'm still waiting for the success case examples applied on a scale that would make any of the predictions come true.

[AnimalMuppet]

You're assuming that frontier AI research can "kick off the exponential" - that there exists such an exponential to kick off, and that frontier AI research can reach it soon, rather than in a couple centuries. Those are massive assumptions that have exactly zero empirical data behind them.

[mikgp]

> How much do these models need to do before people throw their hands in the air and say, ok this is happening.

It's quite interesting to me that your epistemology is basically "God in a box is inevitable" and not skepticism.

> Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).

The blocker is lots of things, but for one? Compute. Millions of instances running parallel, all "probabilistic" testing different theories would require million of GPU clusters.

Two? For subjects like Physics - Atoms. You need to be able to run experiments. You need access to the real world.

Then there's the S curve. Algorithmic breakthroughs operate on orders of magnitude. Eventually we'll run out.

etc. et al.

[bigstrat2003]

> This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.

Yes, if one must assume something it is generally fair to assume that things will continue as they are. Research breakthroughs do happen, but they are not something for which you can predict the timing.

[danaris]

> This assumes that there aren't algorithmic breakthroughs which reduce training/inference costs by several OOMs.

Of course it does.

You cannot operate based on the assumption that you will hit a breakthrough in any given span of time. Breakthroughs are, by their very nature, unpredictable—both on when you will reach them, and on whether they exist to reach at all.

It's also assuming that there isn't a hardware breakthrough that will put us another several OOMs ahead on compute.

It's also assuming that there isn't a military "breakthrough" that leaves all our supply lines either bombed or hostile to us.

When you're forecasting, you can only operate based on what is known and knowable now.

Breakthroughs are unknowable. You cannot rely on them.

[pier25]

The only thing that matters is the economy of it all.

Open AI et al are hemorrhaging absurd amounts of money. It's not clear whether there will ever be a good balance between cost, value, and price.

Lots of companies are already questioning the value they get from LLMs at current prices which are obviously not enough to generate profits.

[skybrian]

It’s no longer true that AI tools primarily get knowledge from their pre-training input data. That gives them a baseline, but nowadays AI chatbots and coding agents routinely assume they need to get up-to-date information in other ways, via web searches and other tool calling.

So I don’t see accuracy declining at least for programming.

[gortok]

> nowadays AI chatbots and coding agents routinely assume they need to get up-to-date information in other ways, via web searches and other tool calling. So I don’t see accuracy declining at least for programming.

How do those chat bots discern that the ‘web searches’ they’re using are returning human generated information only that’s been vetted instead of LLM output?

[WarmWash]

Every lab is already training on synthetic data and has been for years now.

[ako]

They don't need to, they can use tools to validate their assumptions.

[taurath]

If someone has a vibe coded website with bad statistics but shows up on a web search, what tools will it use to check those statistics? How will it know what data it needs to validate? What tools will it use?

[foldr]

Humans face the same problem. So this at least shouldn't make AI perform worse relative to humans, even if AI slop degrades the performance of both over time.

[taurath]

I find it to not be acceptable if AI's trend is to degrade performance of both AI and humans at the same time, that's kinda not the goal, right? Why are we spending money to make us dumber?

[skybrian]

It’s true that they are only as good as their input data, but the same is true if you do your own web searches.

[latexr]

When you do your own web searches, you learn to trust certain signals over others. You learn which sources are trustworthy and which are suspicious. LLMs don’t do that, and they present every information to you as being equally valid.

[taurath]

I know within a second whether the search result I am looking at is obvious AI slop (search for almost any health condition, recipe, esoteric questions etc almost always have slop at the top of the result), but LLMs regularly source those sites as the basis for conclusions.

[kristiandupont]

>I know within a second

I am sorry to be pedantic, but the correct phrasing would be "I think I know within a second", which seems like a pretty important distinction.

[taurath]

Sure and thats fair, there's probably many things that are good enough that I miss, but I'm talking about the most obvious possible websites being used regularly as sources in web search. Identifying poor sourcing or misinformation on the internet and what is a credible source has been a lifelong skill I've had to build from the early days of the web, and in the AI boom its only gotten more necessary to be able to not get taken by hucksters with billion dollar budgets.

[intended]

Not quite.

The difference is in rate of generation.

Today, the ratio of good data to slop has dropped. Previously, you couldn't mass produce websites and have content ready to go at a click.

So you had better hit ratios before, and you have worse hit ratios now. The more labour intensive to create content, you lived in a better world.

Hmm, The better the ratio of creating content to verifying content, the better the environment we live in.

[vitally3643]

The same way you and I do: vibes

Welcome to the postmodern internet. It's vibes all the way down.

[gortok]

Upvoted you, of course; but it’s worse than that. It’s vibes being marketed as correctness. To the lay person (and unfortunately, to more than a few folks who should know better), computers don’t “make up” information. Maybe some good (in some weird way) that comes from all of this is that we stop using LLMs for recitation of facts.

[kristiandupont]

>LLMs-as-AI can only give us probabilistic outputs based on inputs

I am not completely sure what you are saying here, but it sounds like a variation of the "it's just a stochastic parrot" argument, which is reductionist. The human brain is also just a bunch neurons firing.

[gortok]

I’m using the term “stochastic parrot” exactly as the author of the paper did, and incidentally an interview with her was on HN yesterday. https://news.ycombinator.com/item?id=48805401

To wit:

> Another one is that “stochastic parrot” got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way, but that’s not how I intended it, because it’s just a description of what these systems actually are. To see it as an insult requires either the belief that the large language model is the kind of thing that can take offense, which it isn’t, or that these large language models should be understood as steps toward this grand ideal that I don’t hold of artificial intelligence.

> What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.

[y0eswddl]

"i am not sure what you're saying..."

"...but I'm gonna argue against it anyway."

[aitchnyu]

Is model collapse still an unsolved topic? Once saw a method that determines if a training input is high quality.

[tshaddox]

> The problem for us isn’t that AI will take our jobs; it’s that snake-oil salesmen can sell the idea that AI will take our jobs, investors buy into it, companies try it, fire their folks, the snake-oil salesmen IPOs, the companies that bought into this idea implode in some form or fashion, and the salesmen have already taken the money and ran.

Or, it eventually becomes clear to enough people that the AI companies aren't going to make enough money to justify their valuations, so the asset bubble bursts, the economy crashes, and we lose our jobs.

[grttw1]

Many jobs imo have been under threat for many yrs.

The tremendous growth in earnings meant some fake excess head count number was viable.

Google faced an activist investor who practically forced sundar to fire a bunch of people. This is what’s coming for big tech if this AI thing blows up. Apple Is safe because they are clever and saw this from a mile away.

Investors want their cake and they will eat it too.

[catigula]

>As long as the term “AI” means by-and-large LLMs with additional features sprinkled on top, the answer is no. More likely (without careful vetting by the folks aggregating these models) is that the quality will go down as more and more AI-generated output gets subsumed into these models.

Nope. This isn't how it works.

AI progress has largely been synthetic and has produced leaps and bounds capabilities increases in the last couple of years.

Sorry.