[dwc]

Pretty much every conception of AGI is of greater than human intelligence, at least in some aspects.

The reason? Otherwise why build it? Yes, there are cases where AGI as subhuman-level intelligence would have utility, but the remaining cases are far more numerous and have far more utility.

[derefr]

A target of exactly-human intelligence is potentially useful for automating All The Jobs and entering a post-scarcity era without also starting an AI singularity.

Though many approaches to “exactly-human intelligence” would just result in humans that happen to be robots, and we probably don’t want that, especially if the point is to make them do all our work for us.

I don’t know how to refer specifically to the alternative, though—an intelligence that can solve “human-hard” problems, without needing anything like sentience or self-determined goals/desires that would push it toward wanting things orthogonal to its designed purpose. Human-adjacent AGI? Human-competitive Tool AI?

[dwc]

Much of your response seems to orbit around the utility function, which is the devilishly tricky part from the start. Getting that part right is much more important than a specific level of intelligence.

[gonehome]

Not sure why you're being downvoted for bringing up a valid point. Though I'd argue the reason is more that stopping at human level is a hard optimization to hit.

Once there's a way to train a general intelligence it's likely hard to contain the improvement - that's why the goal alignment problem is important. Even the current human architecture running at a billion operations per second instead of its current 100 or so would be a problem.

Human general intelligence is constrained by things like power consumption available via eating or needing a head that can fit out of a birth canal - AGI constraints are a lot less limiting and AGI can improve faster.

[goatlover]

Also by all the other human beings placing limits on what any individual, however smart, can do (i.e. taking over the world). A superhuman AI is going to come into a world full of existing intelligences, some of them machine which may be close to super intelligent already, along with the billions of humans and all their organizations.

[gonehome]

I think the existing intelligences and organizations won't matter, the super intelligence will be a step change like the difference between a human and an ant. If the goals are not aligned up front it'll be a problem.

To put it into perspective if you can run the human architecture on an AGI at a billion operations per second it's like compressing a historical human civilization of learning into a couple hours.

Maybe for some reason this learning process will be slow or maybe there's some reason why it can't scale up quickly, but based on the learning speed in narrow areas that seems unlikely.

There's also the fact that natural selection which is a relatively brute force sexual selection mechanism still results in general problem solving brains being everywhere - it doesn't seem like something particularly rare.

[goatlover]

> run the human architecture on an AGI at a billion operations per second

Maybe a single human, but can you run an entire world civilization and it's environment (basically Earth) in a sped up manner? Because one super fast human-level intelligence is still constrained in a way that all of civilization is not. What is one human sped up a billion times? Is that 1 billion humans? We already have several of those units operating around the clock. And those units have access to the world's resources, whereas one sped-up mind will not, unless it can gain control of the world.

[gonehome]

I said human architecture - so human like ability to solve general problems sped up without the other human constraints (like needing to eat) and focused on solving a problem. It isn't like a billion independent humans just doing random things - maybe if they could all focus and coordinate on a single goal but the communication overhead from that would still make it different in addition to the other biological constraints.

You may not need that much access to the world to learn/infer a lot about it [1] and if a superintelligent AI needed access to more in order to achieve its goal it'd probably be able to get it.

[1] https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...

[gowld]

There are multiple dimensions to scaling. A large number of subhuman-intelligent agents can beat a small number of superhuman-intelligences (and vice versa, depending on specifics), and certainly beats not building anything.

[goatlover]

> but the remaining cases are far more numerous and have far more utility

But why do we want to replace ourselves? Do we just want the machines to do everything for us? Take care of us like we're pets?

[Daemon_Eater]

Different people have different priorities, some subset of people must desire to be taken care of like pets. That said, the relationship does not have to have a master/pet dynamic to it. There are alternatives where ASI takes specific roles away from humanity (like various governmental, administrative, or legal roles), especially those roles where humans consistently fall short and failure has very negative consequences. The driving motivations for ASIs will determine what kind of relationship we have with them. They may all have the potential to lead to mass subjugation but, that isn't set in stone, we could end up as partners in a shared future.

I for one hope they can fortify my deep solipsism with more detail, my own personal paradise Matrix. Or maybe they can just take care of me while I waste away on drugs, ensuring no negative externalities.

[blhack]

There is a value to human life beyond just "how smart you are".

>Otherwise why build it?

Because it would be better to send robots into dangerous situations that it is to send human beings.

[dwc]

This is why I mention that there are specific cases for subhuman levels of intelligence.