[blamestross]

As a distributed systems engineer, we are a LONG way from "magical scalable ai".

The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.

We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.

I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.

Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.

AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.

[ACCount37]

Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be.

And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.

I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.

The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.

Which means: AIs can get good at it too. It's not a wall - it's a skill issue.

[blamestross]

I think you are taking an entire space of "intellectual immune system" out of consideration. More than that, you are ignoring the core reason I think they are bottle-necked. They might have more access to compute. But to compute what? The bottleneck of intelligent behavior isn't compute it is experience. We have a lot of "text encoded experience" to feed it, through our collective corpus of writing, but ultimately potential behaviors can only be tested by active experimentation. No amount of observation can discern correlation from causality. Only active experimentation. The "train on itself" only works in a "toy universe" where the model of consequences are trivial to "test"

So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.

On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.