[ACCount37]

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.

How do we get to ASI? That's what recursive self-improvement is about.

If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.

We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.

[jandrese]

> recursive self-improvement

How is this better than training the next Google by having the bots do Google searches?

I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.

Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.

[ACCount37]

It's closer in principle to training a chess AI by making it play against copies of itself.

Humans and AIs play the same chess, on the same board, under the same rules. Doesn't stop modern AIs from crushing humans at it. In a narrow closed domain, vastly superhuman capability isn't just attainable - it's expected.

There are issues with going to less narrow, more open domains with it. But evolution producing increasingly advanced intelligence is a bit of an existence proof for broad open domain RSI being possible - there is a serious argument that human intelligence has evolved in a "self-play" pressure cooker, via semi-adversarial optimization against other humans. If a continuous improvement path like this exists in non-intelligent space, intelligent design has "copy that" at its floor.

We already know that AIs can come up with Move 37, or use branches of human mathematics against each other in novel ways, or come up with human-unnatural solutions like LLMs solving ARC-AGI-3 puzzles with A* pathfinding and a constraint solver instead of humanlike spatial reasoning. Matter of whether this can be pushed further. And by "whether", I mean "whether it happens in the next few decades", not "whether it's possible".