[ACCount37]

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

[goatlover]

> AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

These are claims about future AI, not actual facts. Part of the counter argument is the world will already be awash in AIs institutions and individuals make use of. An ASI would arise in a world that is already full of formidable intelligences that provide a check on what it can do. This is what happened with the evolution of replicators/life. No species was able to fully dominate the biosphere because there are too many other capable replicators, and there are always tradeoffs in capabilities.

We imagine the possibility of an unrestrained god-like ASI ruling the solar system. But it's just that, an imagination backed by the assumption that self-recursive improvement leads there. Problem is, the real world never turns out to be that simple.

It's probably the case that alien ASI replicators aren't devouring the universe either because of various restraints.

[blamestross]

As a distributed systems engineer, we are a LONG way from "magical scalable ai".

The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.

We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.

I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.

Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.

AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.

[ACCount37]

Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be.

And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments.

I think vision is just a piss poor modality to learn intelligence from? Very low value, per bit and per token both. You only ever want to tap it if you need your AI to operate based on visual data at deployment time. Otherwise, even "experience" is best gathered in text RLVR rollouts.

The secret of human sample efficiency isn't that visual data is somehow better for learning intelligence. It just isn't. Human "training data" is a hundred kinds of awful - humans are just good at scavenging it for all its worth. Evolution has tuned that very well.

Which means: AIs can get good at it too. It's not a wall - it's a skill issue.

[blamestross]

I think you are taking an entire space of "intellectual immune system" out of consideration. More than that, you are ignoring the core reason I think they are bottle-necked. They might have more access to compute. But to compute what? The bottleneck of intelligent behavior isn't compute it is experience. We have a lot of "text encoded experience" to feed it, through our collective corpus of writing, but ultimately potential behaviors can only be tested by active experimentation. No amount of observation can discern correlation from causality. Only active experimentation. The "train on itself" only works in a "toy universe" where the model of consequences are trivial to "test"

So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.

On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.

[garaetjjte]

>the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly.

What proves that AI doesn't have the same limitations? There's only so much computation you can do in given space, and all communication is limited by universal speed limit.

[ACCount37]

What? Humans are made of sloppy wet meat. Brains are nowhere near brushing against the physical limits of computation, speed of causality or others, in any way, fashion or form. You need to put a lot of intelligent design on the table before you even start getting close to those walls.

Which doesn't bode well for the future of human intelligence. Computing hardware gets better at what it does generation to generation, but no one is about to release Human Brain 2.0 any time soon. Human mind is not a fast-moving target.

Principal-agent problem isn't a physical law. It's a limitation that AIs don't have to suffer from. Humans have to delegate to other humans - but for AI, "principal" and "agent" might just be the same exact system instanced twice.