[ACCount37]

It's a fairly mainstream position among the actual AI researchers in the frontier labs.

They disagree on the timelines, the architectures, the exact steps to get there, the severity of risks. Can you get there with modified LLMs by 2030, or would you need to develop novel systems and ride all the way to 2050? Is there a 5% chance of an AI oopsie ending humankind, or a 25% chance? No agreement on that.

But a short line "AGI is possible, powerful and perilous" is something 9 out of 10 of frontier AI researchers at the frontier labs would agree upon.

At which point the question becomes: is it them who are deluded, or is it you?

[afavour]

Sure, when you get rid of the timelines and the methods we'll use to get there, everyone agrees on everything. But at that point it means nothing. Yeah, AGI is possible (say the people who earn a salary based on that being true). Curing all known diseases is possible too. How will we do that? Oh, I don't know. But it's a thing that could possibly happen at some point. Give me some investment cash to do it.

If you claim "AGI is possible" without knowing how we'll actually get there you're just writing science fiction. Which is fine, but I'd really rather we don't bet the economy on it.

[ACCount37]

I could claim "nuclear weapons are possible" in year 1940 without having a concrete plan on how to get there. Just "we'd need a lot of U235 and we need to set it off", with no roadmap: no "how much uranium to get", "how to actually get it", or "how to get the reaction going". Based entirely on what advanced physics knowledge I could have had back then, without having future knowledge or access to cutting edge classified research.

Would not having a complete foolproof step by step plan to obtaining a nuclear bomb somehow make me wrong then?

The so-called "plan" is simply "fund the R&D, and one of the R&D teams will eventually figure it out, and if not, then, at least some of the resources we poured into it would be reusable elsewhere". Because LLMs are already quite useful - and there's no pathway to getting or utilizing AGI that doesn't involve a lot of compute to throw at the problem.

[afavour]

I think you're falling victim to survivorship bias there, or something like it.

In 1940 I might have said "fusion power is possible" based entirely on what advanced psychics knowledge I had. And I would have been correct, according to the laws of physics it is possible. We still don't have it though. When watching Neil Armstrong walk on the moon I might have said "moon colonies are possible", and I'd have been right there too. And yet...

[ACCount37]

Those two things are prevented by economics more than physics.

For AI in particular, the economics currently favor ongoing capability R&D - and even if they didn't favor AI R&D directly (i.e. if ChatGPT and Stable Diffusion never happened), they would still favor making the computational inputs of AI R&D cheaper over time.

Building advanced AIs is becoming easier and cheaper. It's just that the bar of "good enough" has gone off to space, and a "good enough" from 2020 is, nowadays, profoundly unimpressive.

I'm not sure how much does it take to reach AGI. No one is sure of it. But the path there is getting shorter over time, clearly. And LLMs existing, improving and doing what they do makes me assume shorter AGI timelines, and call for a vote of no confidence on human exceptionalism.

[afavour]

> But the path there is getting shorter over time, clearly.

Why do you assume there is no hard limit we’ll hit with the current tech that prevents us from reaching AGI?

[ACCount37]

The first reason is that LLM advancements are yet to stop. And the second reason is that "the current tech" is a moving target.

If, let's say, in 2029, it turns out that autoregressive transformer LLMs have exhausted their potential, the R&D that goes into improving them now would be put into finding alternatives. And I struggle to imagine not finding any.

[AntiDyatlov]

In the case of nuclear weapons, we had a theory that said they were possible. We don't have a theory that says AGI or ASI is possible. It's a big difference.

[adrianN]

There are plenty of people that argue that you need nontechnological pixi dust for intelligence.

[ACCount37]

Yes, quite unfortunately. That reeks to me of wishful thinking.

Maybe that was a sensible thing to think in 1926, when the closest things we had to "an artificial replica of human intelligence" was the automatic telephone exchange and the mechanical adding machine. But knowledge and technology both have advanced since.

Now, we're in 2026, and the list of "things that humans can do but machines can't" has grown quite thin. "Human brain is doing something truly magical" is quite hard to justify on technical merits, and it's the emotional value that makes the idea linger.

[dirkc]

There are also people who think there might be emergent behavior at play that would require extremely high fidelity simulation to achieve.

Also, the real thing (intelligence) as it is currently in operation isn't that well understood

[grayhatter]

> But a short line "AGI is possible, powerful and perilous" is something 9 out of 10 of frontier AI researchers at the frontier labs would agree upon.

> At which point the question becomes: is it them who are deluded, or is it you?

Given the current very asymptotic curve of LLM quality by training, and how most of the recent improvements have been better non LLM harnesses and scaffolding. I don't find the argument that transformer based Generative LLMs are likely to ever reach something these labs would agree is AGI (unless they're also selling it as it)

Then, you can apply the same argument to Natural General Intelligence. Humans can do both impressive and scary stuff.

I'll ignore the made up 5 and 25%, and instead suggest that pragmatic and optimistic/predictive world views don't conflict. You can predict the magic word box you feel like you enjoy is special and important, making it obvious to you AGI is coming. While it also doesn't feel like a given to people unimpressed by it's painfully average output. The problem being the optimism that Transformer LLMs will evolve into AGI requires a break through that the current trend of evidence doesn't support.

Will humans invent AGI? I'd bet it's a near certainty. Is general intelligence impressive and powerful? Absolutely, I mean look, Organic general intelligence invented artificial general intelligence in the future... assuming we don't end civilization with nuclear winter first...

[ACCount37]

Asymptotic? Are we looking at the same curves?

Recent improvements being somehow driven by harnesses and scaffolding rather than training?

With that last bit, I'm confident that you're not in ML, and not even keeping track of the things from what's known to public.

[re-thc]

> But a short line "AGI is possible, powerful and perilous"

> At which point the question becomes: is it them who are deluded, or is it you?

No one. It is always "possible". Ask me 20 years ago after watching a sci-fi movie and I'd say the same.

Just like with software projects estimating time doesn't work reliably for R&D.

We'll still get full self-driving electric cars and robots next year too. This applies every year.

[kaashif]

> We'll still get full self-driving electric cars and robots next year too.

I've taken a Waymo and it seemed pretty self driving.

[re-thc]

Not that 1. Wink.