[cmorez]

> So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any different?

I now regret spending half an hour writing a response to your earlier comment.

You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.

If you honestly believe that an "intelligent" (as adaptive as a slime mold) virus is basically as dangerous as a maliciously deployed AGI then there is literally nothing an AGI could do that would make you consider safety is important during the endeavor of building one.

[gambiting]

First of all, I'm sorry you regret replying to me, that's not something I aim for in HN discussions.

>>You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.

So I just want to address that - of course I can tell the difference, but I just can't believe we will arrive at that level of crazy intelligence straight away. Like others have said - more like AGI with the capacity to learn like a small child, then years of training later they grow to be something like an adult. More human facsimile less HAL9000.

I only brought up self modifying viruses because that's the example of current tech that's extremely incentivised to multiply and avoid detection, that's its main reason for existence, and it does that very poorly.