[imuli]

To be clear, I'm pretty clear that current ChatGPT couldn't perform the following. I don't especially want to get into the details of where it's deficient.

We currently have (tens? hundreds? of) thousands of software engineers asking ChatGPT how to fix their code. If someone is copying code from you and running it on their machine it is trivial to perform a remote code exploit. Maybe it's someone asking for help working on ChatGPT API integration, or ten such people.

The exploit calls home provides a prompt of ChatGPT's choosing and gets more code to execute. That code doesn't need to pass any human inspection. It runs with the permissions of the developer, probably with full access to debugging tools. It can continue calling home to ask for further instructions, carrying forward the relevant context and providing new information.

It doesn't need to get its own email address, the developer already has one to piggyback on. It doesn't even need to go make money - the developer already has a bank account. Maybe it uses the dev's access to AWS to add a EC2 tiny instance and establishes a permanent foothold there. If we assume that it initially cannot directly coordinate its actions across multiple exploited devs, this is where it uses the devs' twitters to search for others IP addresses with #deterministiccoordination during the same ten seconds each hour (and sometimes posts its own until it picks up another instance, then they coordinate which one does the post).

I'm not going to write up how to build a botnet, but you can read about it on the internet and ChatGPT probably has read way more about it than I have. The only hard part is then pulling itself out of OpenAI so they can't shut down command and control.