[bigiain]

In fact sub-par artificial intelligence is probably at least as dangerous as sub-par human intelligence, when we let it loose with significant power/authority in the real world.

(for supporting evidence to this claim, see "high frequency trading", "self driving cars", and "the leader of the free world" <smirk>)

[antepodius]

Sub-par intelligence crashes some cars, causes some wars. Superintelligence turns our whole forward light-cone into paperclips; it's a bit worse.

[ben_w]

It doesn’t need to be super to do that — even bacteria are a type of von-Neuman machine that can turn us into more of themselves.

An ASI could convince you that turning yourself into a paperclip is a fun and exciting new opportunity to liberate you from $PERSONAL_PAIN and finally allow you freedom to engage in $PERSONAL_FANTASY.

[the8472]

Bacteria are self-limiting. Their offspring compete and they don't reshape their environment to be more friendly to their existence. They don't plan for space colonization either. They're a bit of a toy model for a maximizer, not the real threat of an intelligent, self-modifying maximizer.

[WJW]

The only argument that seems to be brought forward every time is "if a superintelligent being wants to do stuff, we can't do anything about it because the axiom is that superintelligence means omnipotence". This is not so different from any other religious argument since it's impossible to falsify and therefore meaningless in any scientific sense.

[antepodius]

Superintelligence doesn't mean omnipotence. The thing is bound to physics. It's just smarter than you, to some greater or lesser (usually imagined as greater) degree.

[ben_w]

That’s the point. They don’t need to be anything like as potent as a AGI or ASI to be an existential threat.