[seanmcdirmid]

It is only a matter of when, like practical nuclear fusion. We know it is possible in theory.

Regardless, that programming is so unaugmented while other expert engineering fields are much better is shameful to the entire profession.

[erikpukinskis]

No, we don’t. Nuclear fusion is a theory, there is evidence it is possible. We can observe it by looking at the stars.

Omnipotent AI is not a theory. There’s no evidence it is possible. I have yet to even see a a falsifiable hypothesis about how it could come about. The most advanced hypotheses are something along the lines of “they’re getting smarter do eventually they’ll be infinity smart” which is not a strong claim.

Evidence suggests intelligence is niche specific. Humans are smart, but in the middle of the ocean a jellyfish is smarter. A better prediction about AI is they will surpass us in some ways and not others.

[seanmcdirmid]

> Omnipotent AI is not a theory.

That's not what was suggested. Human-level intelligence is sufficient to cause a singularity (because then they can start improving their own programs).

> better prediction about AI is they will surpass us in some ways and not others.

That isn't a better prediction at all. Regardless, it is uninteresting to my original point, which is we want to live in the Iron Man, not Ultron, phase now anyways.

[erikpukinskis]

> Human-level intelligence is sufficient to cause a singularity (because then they can start improving their own programs)

Your assumption is that human-level intelligence is bound by offline simulation capacity.

If humans are already optimally utilizing offline simulation (in the biological world we call that imagination) then the “human level” AI will have just the same limitations as a human.

That’s the counter-proposal you have to make falsifiable for your guess to become a hypothesis:

That any human-level AI, in order to become human level, will be bound to the same interactive constraints on learning that humans are.

Think about World War II for example: were strategies limited by offline simulation capacity, or were they limited by the fact that you can only try (and therefore sample consequences for) one at a time?

I’m not making a claim here either way: I’m just saying you wave this whole debate away by saying “human intelligence plus unlimited simulation equals superhuman intelligence”

I’m not saying your wrong, it’s an interesting thought experiment: I’m just saying not only is there zero evidence for that, there’s not even (to my knowledge) a robust model describing the mechanism by which that could expect to be true.