[snewman]

[author here]

> And AGI doesn't need to completely destroy the earth to be really bad for humans. Just taking over a lot of the resources we need would do the trick.

Agreed. I am not arguing that AI could not become superhuman, or could not overwhelm us. I'm merely arguing that:

1. It is not guaranteed that mildly superhuman AGI would inexorably lead to a runaway feedback loop of capabilities increase. I'd agree that it's possible, but I often see statements that it is inevitable, because an AI smarter than us would be able to create an AI smarter than we can. Such statements fail to take into account that the sequence of successive AIs might converge (at mildly superhuman) rather than diverging (to a singularity).

2. Even if AI capabilities diverge, there's no guarantee that would happen quickly ("foom"), because even as AI capabilities increase, the effort needed to achieve each further increment in capability will almost certainly also be increasing.

I do take the potential for superintelligence very seriously; see, for instance, https://amistrongeryet.substack.com/p/get-ready-for-ai-to-ou.... But I also think that some current discussion is going overboard in the other direction, and waving away likely hurdles on the path to superintelligence (see https://amistrongeryet.substack.com/p/the-ai-progress-parado...).

[gremlinsinc]

If I had the means, and will - I could easily see myself building an AI that trains future generations better/faster than the last, does this weekly, and also manages a silicon factory that recycles old ai machines into new ones, or old gpu's into new ones, or other chips and continually explores new/better hardware options.

All self-contained with very little human oversight, similar to how humans can reproduce, pass on genes, and then do it all over again until we evolve new traits, AI systems could do the very same thing.

I can see this scenario where language models are evolving weekly or even doubling in abilities, is possible this decade.

Running out of materials? AI drones can setup bases on the moon (less gravity issues) to go out and mine asteroids, bring it back to the moon to build more processing power. The entire moon could be turned into one mega structure super computer that could house an ai that helps manage all the things humanity needs it for. Traversing into scifi territory, but I thought where we are now was nearly impossible by now. LLM's and Stable Diffusion, and coming multi-modal modals are going to affect every aspect of life, one way or another whether it gets to full-super-intelligence doesn't matter, it will still change everything.

[danaris]

There's a lot of handwaving in there.

We don't currently have completely automated computer factories. Large chunks of manufacturing are automated, sure, but a) those automated segments are monitored and controlled by humans, and b) there are also significant parts of the assembly process that are fully manual.

Does that mean that fully automated construction is impossible? Of course not. But it does mean it's not something we are certain we can do today, without additional genuine breakthroughs.

Furthermore, what you're proposing is not merely automated construction, but automated improvements to the construction process. That means you need to be able to reconfigure the entire manufacturing process without human intervention. I feel reasonably confident in saying this is not feasible with our current technology.

Beyond that, your very first comment—about needing the means—is very much nontrivial. I don't recall the exact figures, but I was definitely seeing stories about ChatGPT being massively expensive to run, both in terms of money and in terms of energy. And that's just a current-generation LLM. Attempting to bootstrap from there to....the singularity, I guess? is likely to take more energy than is feasible to dedicate to any such project even if it were possible.

And finally, you say the AI will "train future generations"—train on what? An LLM's quality is always going to be largely dictated by its training data. How is an LLM going to be able to train a next-generation LLM any better than it was trained, especially, again, without human intervention? It's nearly impossible for what you're describing to result in anything that can do useful work, simply because training like that requires humans in the loop giving feedback at every step.

[AstralStorm]

Unless that AI breaks speed of light, the distances to acquire additional resources will be relatively prohibitive.

The moon lacks a lot of rare earth minerals required to make modern transistors, for example. The AI would have to set up shop in asteroid belt or Kuiper belt or Oort cloud, and except for the last one it limits available energy by quite a lot.

[Nevermark]

I think the writing is on the wall.

There don't seem to be any arguments beyond the trivial observation that things could slow down. But they haven't slowed down.

New models keep surpassing us (in some cases the whole human race) dramatically in new areas of greater generality, while their "weaknesses" also improve dramatically.

It's hard to imagine where a fully multi-modal model, with long context, and a sense of information confidence, and ability to manage its own notes/whiteboard information, will not exceed us. On top of improving in all the areas it already outdoes us.

People are working on each of those improvements right now. (By fully multi-modal I mean text, audio, image, video, simulated and real physics, touch, motor control, team communication, software/internet access, and whatever other senses or decision forms are helpful.)

I don't see a general AI vastly smarter than any one of us, or all of us together, taking longer than 2030-2033 time frame.

[danaris]

> But they haven't slowed down.

But...they have.

How many stories have there been about how GPT-3 and GPT-4 have gotten worse over the period they've been out?

And it's not like we got GPT-3, and then GPT-4, and then, within the same time frame, GPT-5 with a similar increase in quality.

Sure, people are working on more improvements, but they're not here yet, and while that doesn't mean they will never come, it does mean that, compared to what appeared to be a rapid rush of "AI" progress, things have slowed down.

"Progress is still being made" and "progress has slowed down compared to the speed that generated all the hype" are not incompatible statements.

[Nevermark]

> But...they have. How many stories have there been about how GPT-3 and GPT-4 have gotten worse over the period they've been out?

They are experimenting with trade offs. In this case, things like appropriateness, less overconfidence, etc.

That isn’t indicative of any slowdown.

> Sure, people are working on more improvements, but they're not here yet, and while that doesn't mean they will never come, it does mean that, compared to what appeared to be a rapid rush of "AI" progress, things have slowed down.

I don’t follow.

People and teams keep publishing and releasing new techniques but big projects don’t release major updates as quickly.

How are your arguments more relevant to GPT4 than they would have been for GPT3? Earlier models?

[danaris]

You really seem to have missed the main point of my post.

Yes, people are still working on stuff. I never said they weren't. I never said they were working any less hard than they had been.

But progress has slowed down from the leaps that it took over the past couple of years to get us to GPT-3 and GPT-4. Whatever "new models" are doing in September 2023, they are absolutely not making the same clear advances that we saw previously. They are incremental improvements. Which is good! It's important! It's progress! But it's unquestionably slower than the breakthrough that led to this generation of LLMs.

Now, it may be that that wasn't what you were thinking of when you said things haven't slowed down, but that was absolutely not clear from your post. And there have been enough people trumpeting loudly that the pace of progress would continue exactly as fast as it was before—breakthrough after breakthrough, leading rapidly to GPT-5 and beyond, and causing millions of white-collar jobs to be automated—that to simply say, without qualification, "things haven't slowed down" is, at best, ignorant of the way it will be taken by many.

[Nevermark]

> But progress has slowed down from the leaps that it took over the past couple of years to get us to GPT-3 and GPT-4.

[emphasis mine]

GPT-4 was only released March 14, 2023 [0], so until March 14, 2025 you don’t have any data to support your claim.

Elsewhere models continue to update quickly. While we wait for GPT-5, much smaller models keep getting better.

There have been few times in history where for over a decade such a widely impactful tech has kept improving in stunning steps.

There is no evidence of any slow down.

[0] https://en.m.wikipedia.org/wiki/GPT-4