[ryanjshaw]

This will almost certainly happen, and it will be a terrible mistake - at least in the story I'm working on :)

My theory is that AI generated code will probably look and grow organically (the irony!). Humans will set out requirements, the AI will collate these into a series of tests, and it won't care how neat or understandable the code is, provided the tests pass. Basically an extremely diligent junior developer.

There will be efforts, probably in the open source world, to produce AIs that tidy up things by structuring the code sensibly, eliminating dead code, etc. Maybe even some effort to pass laws around standards and limits on what AIs have access to when involved in certain industries, for example, no external communications. But, in the name of efficiency, enterprise developers will be forced to use something that merely pays lip service to all of this.

Eventually nobody will have any clue what code is running and what it's actually doing. We may even lose the tools and access we need to perform those inspections. And that is when the AIs will coordinate their attack.

[apatil]

Sounds like an interesting story! I would love to see more sci-fi that really tackles AGI. I used to love sci-fi but most of it, even "hard" sci-fi, has become unwatchable or unreadable for me because the limited role of AGI is such an all-consuming plot hole.

[tsunamifury]

The idea that the AI would attack us once it reached that level is like saying humans would attack ants. It would simply be entirely indifferent towards us and mow us over by accident at best.

We won’t live in a human centric universe because power will express itself in a new species.

[ryanjshaw]

Humans regularly eradicate ants that overstep their bounds, and that's what I see happening with humans and AI.

We will be allowed to do our thing - provided we don't get too bold.

There will be some who are content with this arrangement, and some who aren't, and that tension is where I believe a story can be told.

[marviel]

I really hope we can figure it out. It's either we figure out alignment, we regulate ourselves VERY VERY CAREFULLY, or we perish -- I think?