[fragmede]

If you're gonna look at it that way, then the halting problem is just a dressed up computer science version of the question "can God make a rock so heavy even he can't lift it?" Could an infinitely smart AGI come up with the answer to the unanswerable question?

It doesn't need to be infinitely smart to do a better job than the worst of humanity's blunders.

[kbrkbr]

But isn't on the other hand the current AGI problem posed similar to the question "Are you not afraid that genetic engineers grow babies with bigger and bigger brains?" We don't know if that won't break down somewhere. Looking at different examples it probably will. Scaling things infinitely is a pure math only concept it seems.

[smaudet]

Again, infinity is illustrative.

Put another way, you do not deny a proof by inference because it "leads to large numbers".

We don't need to speculate, we can see many, many examples today of more and less intelligent species, and also what happens to the less intelligent ones, even taking humans out of the equation.

I'd argue, even a machine intelligence that merely managed to be mildly smarter than us would be a threat. AGI merely has the potential to be infinitely smarter than us, but that's somewhat irrelevant given we might not even be smart enough to realize how much smarter they are (a cat will likely not appreciate the difference in intelligence between a dog, an elephant, or a dolphin, despite all those animals being generally smarter).