[smaudet]

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.

The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).

An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.

Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.

That's hand waving.

[kbrkbr]

From my point of view we have simply no idea what a infinitely smart AGI is and how to build it.

How would it make the combinatorial explosion in state space search go away, to pick one example?

And if it doesn't, is it then an infinitely smart AGI?

The concept seems to assume all problems humans struggle with can be solved. The halting problem is one witness that this is probably not true.

[fragmede]

If you're gonna look at it that way, then the halting problem is just a dressed up computer science version of the question "can God make a rock so heavy even he can't lift it?" Could an infinitely smart AGI come up with the answer to the unanswerable question?

It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.

[kbrkbr]

But isn't on the other hand the current AGI problem posed similar to the question "Are you not afraid that genetic engineers grow babies with bigger and bigger brains?" We don't know if that won't break down somewhere. Looking at different examples it probably will. Scaling things infinitely is a pure math only concept it seems.

[smaudet]

Again, infinity is illustrative.

Put another way, you do not deny a proof by inference because it "leads to large numbers".

We don't need to speculate, we can see many, many examples today of more and less intelligent species, and also what happens to the less intelligent ones, even taking humans out of the equation.

I'd argue, even a machine intelligence that merely managed to be mildly smarter than us would be a threat. AGI merely has the potential to be infinitely smarter than us, but that's somewhat irrelevant given we might not even be smart enough to realize how much smarter they are (a cat will likely not appreciate the difference in intelligence between a dog, an elephant, or a dolphin, despite all those animals being generally smarter).

[goatlover]

Infinities aren't a physical reality. Resource are always limited, physics is limited at the Planck scale. You can only do so much compute in a finite volume of space, and there will only be so much energy available.

As for simple needs, humans also have complex ones around social interactions and the need for mental stimulation.

[smaudet]

Infinity is....subtle.

They are more characterized by how they grow and when they stop than by their "physical reality". Proof of this is in that different infinities exist - characterized precisely by how fast they grow, i.e., one "infinity" is "larger" than the other.

My point is, getting hung up on "infinity" as being unrealistic is not the point. It is the tool with which to understand how thing behave. The same as any calculus problem - you take the limit to the infinity to understand how the function behaves.

[JumpCrisscross]

> We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

That effect is underwritten by economic demand and moderated by economic costs. There are more reasons to expect the trend to asymptote than somehow turn into an infinite process.