[Nyubis]

>Have you ever programmed any piece of software that suddenly implemented features of its own?

I'm not sure what particular fear mongers you're talking about, but Bostrom and the like are talking about software that's specifically made to change and improve itself on a fundamental level.

[nojvek]

This is no different than Facebook ads/news feed adapting to your interests and only showing what you like in the hope of maximizing total engagement and retention.

We all know that Google/Facebook/Bing build a little bubble for you and customize results and ads based on your behaviour.

[erikbye]

I'm saying that's the pipe dream. Software won't change on its own.

[Nyubis]

We already have software that can change on its own. Neural nets are software too, and with things like backpropagation they can update their weights, essentially changing themselves.

I'm not saying that this level of change is enough to get the disaster scenarios that Bostrom talks about, but it's a folly to say that self-changing software can't possibly exist.

[p1esk]

Neural net changing its weights is no different from any other software updating the values of its variables.