I'm not saying it will never work, never is a long time, but it's not the perfect domain for AI. It's kind of the opposite, actually.
First and most importantly, programming computers is a very precise endeavor. The logic needs to be exactly correct, not only statistically correct. 'Close enough' won't cut it, not even remotely, and not even for relatively unimportant software.
Second, the general problem is undecidable. This isn't a roadblock per se, because we are reasonably good at other undecidable problems (viz. garbage collection), but it means simple algorithmical approaches won't work.
Third, software is in a weird place because it requires working at different abstraction levels simultaneously. Often, top-level specifications are fuzzy and incomplete, but some parts require absolute precision and we need to "drop down" to a lower level of abstraction. Humans are able to make the process work (kinda) using lots of common sense, something machines are currently very bad at.
If you require the operator of the 'AI' to fill in the details, you just invented a very complicated compiler.
Finally, rarely if ever present-day software is made once and never changed: the output needs to be inspectable and maintainable, other software might need to call into it, etc.
If the pipeline is more complicated than the software itself, I might as well be writing the code myself.
I can see some minor, specific tasks being increasingly done with AI, and I can see tools making more and more use of AI technology, but AI generated code isn't even on the horizon.
It seems like many of those issues would be solved by having a human write a comprehensive test suite, and then the AI would write an implementation that can pass all of the tests.
Eventually you could also have another separate AI that learns to generate the test suite itself from instructions given to it by a human (or even another AI!).
Most program implementations would not be perfect, but as we know, software written by humans certainly isn’t perfect either.
First and most importantly, programming computers is a very precise endeavor. The logic needs to be exactly correct, not only statistically correct. 'Close enough' won't cut it, not even remotely, and not even for relatively unimportant software.
Second, the general problem is undecidable. This isn't a roadblock per se, because we are reasonably good at other undecidable problems (viz. garbage collection), but it means simple algorithmical approaches won't work.
Third, software is in a weird place because it requires working at different abstraction levels simultaneously. Often, top-level specifications are fuzzy and incomplete, but some parts require absolute precision and we need to "drop down" to a lower level of abstraction. Humans are able to make the process work (kinda) using lots of common sense, something machines are currently very bad at. If you require the operator of the 'AI' to fill in the details, you just invented a very complicated compiler.
Finally, rarely if ever present-day software is made once and never changed: the output needs to be inspectable and maintainable, other software might need to call into it, etc. If the pipeline is more complicated than the software itself, I might as well be writing the code myself.
I can see some minor, specific tasks being increasingly done with AI, and I can see tools making more and more use of AI technology, but AI generated code isn't even on the horizon.