[Nyubis]

We already have software that can change on its own. Neural nets are software too, and with things like backpropagation they can update their weights, essentially changing themselves.

I'm not saying that this level of change is enough to get the disaster scenarios that Bostrom talks about, but it's a folly to say that self-changing software can't possibly exist.

[p1esk]

Neural net changing its weights is no different from any other software updating the values of its variables.