[ivalm]

And the monkeys thought, “how could a human be dangerous?” “Would they clobber us with stones? Surely we are stronger!”

The problem is that the toolset available to someone who is much more intellectually capable is beyond what we can think of.

People are not afraid of AGI because it will behave like a very smart human, people are afraid of AGI because the capability gap will be more like the one we experience between humans and other animals.

The toolset available to humans when dealing with monkeys is literally incomprehensible to the monkeys.

The toolset available to AGI is similarly incomprehensible to humans.

[uh_uh]

Part of the issue is that taking AI alignment seriously does require some level of intellectual humility — a quality that the HN comment section famously lacks.