[logiduck]

We may have the tools to manage it, but we are losing the ability to understand it.

AI writing software will be a exponential explosion in software complexity.

AI would very well create its own programming language to be more efficient for the AI to code in that we have no hopes of understanding. Imagine that AI started to output a large SaaS app written today in Python in Assembly because for the AI the extra cognitive overhead of using Assembly doesn't exist. At first we might resist and tell it to use a language we understand, but then as time goes on we grow more comfortable that it does the "right" thing and down the road people are just generated raw Assembly using AI without really understanding what the code is doing, only looking to see if the code behaves the way they expect.

Imagine entire codebases spun up in seconds with so many lines of code, a single person would never have hopes of understanding everything, needing to rely on AI to summarize and explain the code for them. Now imagine that massive code base being iterated and worked on for a decade over the life of a company.

AI could bring Terabyte sized code bases in a decade or so.

[throwuwu]

This already happens when you compile code for a specific architecture. The thing you fear will happen has already happened and we’re fine.

I’d worry more about generated code from an AI that doesn’t fully understand the codebase. That would be as bad as letting the junior devs run loose.

[logiduck]

Not really at all.

What you are describing is an traceable transformation of code in which there are several intermediate layers that people can inspect and understand. They can inspect the exact rules for that transformation. The process is repeatable and verifiable.

What I am describing is a black-box stochastic generation of low level code in which there is no higher level representation anymore. AI generating Assembly not by a set of rules, but using statistics. There will be no individual layers to unwind or inspect, because for AI it doesn't need them. Our separation of concerns was built for our human brains and limiting complexity of projects to our understanding.

[throwuwu]

If you have an AI capable of writing machine code based on natural language you likely also have an AI that can translate that machine code to any other language you would like. You could then use a normal compiler to verify if it is correct and then read the code yourself. Or you could just get good at reading and writing assembly.

[logiduck]

Yes, that is exactly my point. You will have to rely on AI to translate it back out, but that translation is built on probabilities not machine rule-based translation. So you can ask and have the AI explain everything to you, but you are still trusting the "black box" to tell you what is happening. Very different from today.

Also, you can "get good" at reading assembly, but that doesn't matter if the AI can output a custom OS from scratch and a custom VM to execute the program it wrote to solve your use case. It will be so impossibly complex that it would be the equivalent studying protein folding.

Instead people will just trust the AI.

It also won't help you if the code base the AI produces for a SaaS app is a million lines of assembly.

Instead of having different layers of OS, compiler, high level language, an AI will just be able to produce one layer. because after decades of trusting the AI to write our code, why wouldn't it?

The current gen of AI outputting code that in human-centric programming languages will be a blip in the history of AI. As it advances, it can just skip that step.

Its will be orders of magnitude more complex and opaque than anything we have today.