[ekidd]

The author is completely right about the AI Lab's promised vision of the world: They claim to want to create superhuman intelligence, which will produce vast abundance. But superhuman intelligence would be extremely dangerous, so it needs to be controlled by a tiny "priesthood" of trusted people, or somehow designed so that the superhuman intelligence could be trusted. (We have no idea how to do that.)

But the author's vision is also suspect, if you assume that the models will become much more intelligent:

1. Hypothetically, we can't give every human their own personal SkyNet to command. That would, uh, probably end very badly. If everyone gets an agent, those agents can't be too capable?

2. If you do somehow build a model that's much smarter than you, what do you contribute by managing it? How many people here have ever worked for a well-intentioned manager who couldn't understand the people they managed? So in this scenario, human management would be mostly displaced by agent management. Most companies could lay almost everyone off and let the agents manage each other. We only need humans to manage models now because the models are still pretty broken.

3. If we create models that can genuinely replace humans at almost any task, you won't be able to buy those on the API. At that point, the billionaires and the politicians wouldn't need human workers any more, because everything can be done better using their pet agents. Just have the robots build stuff for the billionaires directly. And if any of the former human peons get upset about being locked out of the economy to starve, then have the agents pilot the drones, too.

Basically, almost none of the people imagining a future of superhuman intelligences have actually though through how it would actually work in the real world. We're going to spend trillions of dollars and vast amounts of resources chasing the goal of making ordinary humans obsolete. Now, that goal might be unobtainable, I hope. But I'm deeply alarmed at how much we're spending pursuing it.

[fragmede]

I don't take 3 as a given. There's just too much going on in the space for one cloistered company to control it all and be in control of it. Today, I can run a local LLM with tool calling as a weirdo nerd that is able to accomplish a small subset of tasks a human could do. There are enough weirdo nerds out there that I don't see a future where future, more capable versions will be exclusively locked down and restricted to a blessed cabal. It won't be available to everyone for cheap, but that's not the same thing as being locked up and only available to a blessed priest hood. As long as HuggingFace is up, I'm not worried about politicians and billionaires being the only ones with a seat at the table.

[ekidd]

> I don't take 3 as a given. There's just too much going on in the space for one cloistered company to control it all and be in control of it.

Yeah, in my comment, I was assuming the publicly-stated goals of the labs actually came true. I assumed that they achieved true AGI (defined as "about as smart as Fable, except it can manage long-term tasks as well as a smart human, too"), and that this would cause widespread unemployment and fully-automated production. And furthermore, I assumed that once they could use AGI to automate more AI research, they could use that to make even smarter models. The labs call this "recursive self-improvement" (RSI) and like to insist it will happen Real Soon Now.

Given this chain of events come true, then I think there's an excellent chance that the resulting models will never be sold via an API. Given those assumptions, the labs would probably make more money by keeping all the compute to themselves and just ordering the AI to start and manage businesses. Imagine Anthropic having an in-house OpenClaw that could plan and run a successful startup with no human input besides an annual "board meeting" with the humans.

This isn't the only possible outcome, of course.

- Maybe the labs are wrong and progress stalls out well short of superintelligence. This would be nice!

- Maybe true AGI is only requires one or two clever algorithmic tricks beyond what we have now. In this case, training costs might go down, and superhuman models might become widespread. I suspect that possible future would be extra weird.