[tenthirtyam]

My expectation is that there'll never be a single bust-up moment, no line-in-the-sand beyond which we'll be able to say "it doesn't work anymore."

Instead agent written code will get more and more complex, requiring more and more tokens (& NPU/GPU/RAM) to create/review/debug/modify, and will rapidly pass beyond any hope of a human understanding even for relatively simple projects (e.g. such as a banking app on your phone).

I wonder, however, whether the complexity will grow slower or faster than Moore's law and our collective ability to feed the AIs.

[layer8]

Maybe software systems will become more like biological organisms. Huge complexity with parts bordering on chaos, but still working reasonably well most of the time, until entropy takes its course.

[leptons]

It's already like that, for a long time. Humans are quite capable of creating complex systems that become unwieldy the bigger they get. No one person can understand all of it. I will offer the AT&T billing system as an example that I'm all too familiar with as a customer, due to the pain it causes me. So many ridiculous problems with that system, it's been around a long time, and it is just so screwball.

[layer8]

Biological systems are vastly more complex, and less brittle, in the sense that killing a single cell doesn't cause system failure like for example removing an object file from a program often would. Just look at the complexity of a single sell, and try to imagine what an analog of similar complexity would be in software.

[leptons]

You're kind of jumping around in scope here, and I think you got it a little wrong.

>Biological systems are vastly more complex, and less brittle, in the sense that killing a single cell doesn't cause system failure like for example removing an object file from a program often would.

Removing a single node from a massively parallel system doesn't kill the system either, it only removes one node, another node will spin up and replace it, just like a failing cell would in a biological system. One liver cell doesn't do anything for the host on its own, it's part of a massively parallel system.

> Just look at the complexity of a single sell, and try to imagine what an analog of similar complexity would be in software.

Removing some "parts" from a cell certainly could kill it, or a million other things that lead to apoptosis, or turn it cancerous. But "parts" isn't analogous to software, DNA is. The same goes for a single node in a system - remove part of the code and you're going to have big problems - for that node. But probably won't kill the other nodes in the system (though a virus could).

There are 3 Billion base pairs in Human DNA. I could imagine that there's more than 3 billion lines of code running important things throughout the world right now. Maybe even in one system, or likely soon there will be. With "AI" doing the coding that number is going to explode, without anyone able to understand all of it. And so I could imagine that "AI" will probably lead to some kind of "digital cancer", the same way there are viruses and other analogues to biological systems.

[tenthirtyam]

> "AI" will probably lead to some kind of "digital cancer"

Gosh I've just imagined someone asking an AI agent to code a computer virus to infect software "X". The virus' code will be wonderfully complex and therefore so will the response of the AI responsible for keeping "X" uninfected and in good working order.

I was imagining code becoming awesomely complex even without the adversarial element in play.

[tenthirtyam]

Replying to myself here. Maybe coding will eventually be simply learning how to give an AI the right prompt. e.g. instead of

  "Hey AI, create my new banking app with such-and-such functionality, appearance,  properties, APIs, network connections etc"

we will instead do:

  "Hey AI, you are a banking app on a user's cellphone. Connect to mybank.com, authenticate the user and allow the user to perform these-and-those actions in a sensible interface in accordance with the API spec. Don't let yourself be jailbroken."

Then the virus writer's job changes into jailbreaking the AI. Obviously with an AI's assistance...?

Then it would be logical to have a single AI on the phone managing all the prompts in parallel: e.g. "Hey AI, be android by doing [actions]", "Hey AI, be firefox...", "Hey AI, be snapchat...", "Hey AI, be [insert app name]...".