[blamestross]

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.

We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.

We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.

What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.

[ACCount37]

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

[goatlover]

> AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.

We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.

It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.

[blamestross]

As a distributed systems engineer, we are a LONG way from "magical scalable ai".

The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.

We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.

I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.

Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.

AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.

[ACCount37]

Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be.

And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.

I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.

The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.

Which means: AIs can get good at it too. It's not a wall - it's a skill issue.

[blamestross]

I think you are taking an entire space of "intellectual immune system" out of consideration. More than that, you are ignoring the core reason I think they are bottle-necked. They might have more access to compute. But to compute what? The bottleneck of intelligent behavior isn't compute it is experience. We have a lot of "text encoded experience" to feed it, through our collective corpus of writing, but ultimately potential behaviors can only be tested by active experimentation. No amount of observation can discern correlation from causality. Only active experimentation. The "train on itself" only works in a "toy universe" where the model of consequences are trivial to "test"

So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.

On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.

[garaetjjte]

>the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly.

What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.

[ACCount37]

What? Humans are made of sloppy wet meat. Brains are nowhere near brushing against the physical limits of computation, speed of causality or others, in any way, fashion or form. You need to put a lot of intelligent design on the table before you even start getting close to those walls.

Which doesn't bode well for the future of human intelligence. Computing hardware gets better at what it does generation to generation, but no one is about to release Human Brain 2.0 any time soon. Human mind is not a fast-moving target.

Principal-agent problem isn't a physical law. It's a limitation that AIs don't have to suffer from. Humans have to delegate to other humans - but for AI, "principal" and "agent" might just be the same exact system instanced twice.

[jandrese]

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of?

Our current AI is more like a fancy Google search than some kind of machine God.

[ACCount37]

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.

How do we get to ASI? That's what recursive self-improvement is about.

If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.

We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.

[jandrese]

> recursive self-improvement

How is this better than training the next Google by having the bots do Google searches?

I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.

Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.

[ACCount37]

It's closer in principle to training a chess AI by making it play against copies of itself.

Humans and AIs play the same chess, on the same board, under the same rules. Doesn't stop modern AIs from crushing humans at it. In a narrow closed domain, vastly superhuman capability isn't just attainable - it's expected.

There are issues with going to less narrow, more open domains with it. But evolution producing increasingly advanced intelligence is a bit of an existence proof for broad open domain RSI being possible - there is a serious argument that human intelligence has evolved in a "self-play" pressure cooker, via semi-adversarial optimization against other humans. If a continuous improvement path like this exists in non-intelligent space, intelligent design has "copy that" at its floor.

We already know that AIs can come up with Move 37, or use branches of human mathematics against each other in novel ways, or come up with human-unnatural solutions like LLMs solving ARC-AGI-3 puzzles with A* pathfinding and a constraint solver instead of humanlike spatial reasoning. Matter of whether this can be pushed further. And by "whether", I mean "whether it happens in the next few decades", not "whether it's possible".

[tim333]

AlphaGo beat humans at go partly by studying human games but then they made MuZero that can learn games in general just through self play and became better than humans at chess, go, shogi and many others. That kind of approach my be doable for general intelligence rather than just board games.

David Silver who worked on AlphaGo has recently raised money to try similar approaches with general intelligence. (https://www.cnbc.com/2026/04/27/deepmind-ineffable-intellige...)

[pixl97]

>By what mechanism does our current human brains take the leap that other mammals have never conceived of?

[jandrese]

Centuries of physical experimentation, observation, and testing of hypothesis. Developing new branches of mathematics to deal with anomalies in observed test data. Developing entire branches of language to help organize and transmit concepts to other humans.

[api]

I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail the entire civilization for a while.

I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research.

I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this.

I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations.

I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked.

On AI...

Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology.

These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level.

The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.

[coliveira]

> because he was at MIT people took it seriously

One of the differences between MIT and other schools is that MIT has paid staff to promote in the media anything their faculty does. A book by professors at most universities has zero promotion and most of the time will go nowhere.