[lurker616]

How about a simple example - train a model on outputting code which:

- scans open network ports through google dns

- copies itself and the model to that location

- whenever it detects a sigkill or file delete, scan again and save itself somewhere else

Now you've got an autonomous virus that will never die. Add any other power-seeking and self-replicating strategies using AI modules - the task could be "generate code to run as many duplicate processes as possible" or "generate code to find other floating AI modules like me and share model weights with them". Codex might not be there yet but we're getting close.

[polygamous_bat]

Can I tell you something secret? The "AGI" of the form you are describing is already here! Not only that, it's been here since 1988! Crazy right? Just check out this link if you don't believe me: https://en.wikipedia.org/wiki/Morris_worm

(Snark aside, I've realized this type of interactions have always resulted in a very low perception of EA for me. I don't come out of these conversations thinking "what a clueless thing to say by this person who also happened to support EA", I think "wow, EA is full with looney bins who don't know what they are talking about." If you are really into giving what you can and aren't sure why EA gets a bad rep, look no further.)

[lurker616]

I've heard of some of the famous viruses which are obviously more complicated than what I mentioned. But that's the difference - they are hardcoded to perform these particular steps. I'm not saying AGI is the code that does this. AGI is one level higher - a more meta objective - given the objective of "accumulating power", it will learn lower level objectives like "replicating and spreading", which leads to the final output like this code.

Basically, AGI could unleash a thousand "morris worms" - and that would only be for the objective of replicating and spreading. There could be other objectives as well.