[austinjp]

> a paperclip factory isn't self-reproducing (that would be a paperclip factory factory)

This seems an odd refutation for several reasons.

First, the paperclip AI might determine that self-reproducing factories would be an optimisation, and aim to achieve that by any means necessary.

Second, a single paperclip factory that doesn't reproduce might still develop the means of bringing raw materials to it.

Either way, an all-consuming paperclip AI emerges.

In general, I find the equating of the paperclip problem with a religious cult to be naive.

[falcor84]

Agreed, and being a bit of a religiophobe, the thought of living through some sort of butlerian jihad scares me enough, regardless of whether the machines can actually kill us all.