Unfortunately the tests can't (usually) cover all behaviours.
And perhaps more importantly, they don't capture more abstract properties of the code like maintainability.
AI works best in well maintained code, but unless care is taken, the AI will (today, anyway) make the code less well maintained as it goes.
If the AI is allowed to introduce mess into the codebase faster than its ability to deal with the mess increases, the codebase will eventually run into a problem that it's hard to recover from.
And that's kind of the point, some manual testing is always needed - even when humans write the code.
But you can test things and behaviours to a sufficient degree. Nobody really does 100% test coverage on real world code (outside of medical, airplanes and space of course).
Why? How far do you take this, do you inspect the inside of every single thing and program you use?
Your microwave promises to heat your food at 800W for 1 minute, do you open it up and measure the magnetometer strength? Do you double-check the timer that it's accurate?
You want a program that takes in A and returns B within X milliseconds, I deliver it to you. Do you spend the time going through the whole program line by line to see how it was made or do you trust it?
Or when you buy a car do you take it to a garage and dismantle it to see whether it was correctly assembled in the factory? Or do you trust them?
I think the difference here is that 'passing all the tests' isn't the same as satisfying the contract that the consumer is paying for. Even the best test suites won't cover all of the workflows a user might take.
Being pedantic here, but the two scenarios mentioned have regulations as well with repercussions for those products not satisfying their base requirements, whereas a lot of software is the wild-west (I won't be prosecuted for my software not actually working, in most cases).
But what the customer is paying for is defined somewhere, right? Preferably in writing.
You can give that to an agent and have it confirm the application fulfills the spec. Just like a human would have to.
And if the specific test can’t be automated, why? Is it impossible to automate or just inconvenient?
Granted there are immaterial things that we can’t automatically test yet like how the application feels to use. “Snappy” or “smooth” can’t really be quantified with code.
>You want a program that takes in A and returns B within X milliseconds
No, I actually want a lot more than that. I just didn't communicate all of it, because I'm accustomed to you understanding what good software is. (Or I didn't hire you in the first place, because I don't trust others to care like I do.)
If the microwave doesn't cook my food well, I can probably adjust for it; and anyway the objective power metric isn't the thing that matters to me, but the temperature and taste of the food I pull out. The code is an entirely different story, assuming I'm actually asking for code and not a finished program. Because I'm going to have to integrate it into something else.
And yeah, like the sibling comment said, the currently existing tests aren't actually the full interface. And they, too, are code, which can be wrong.
And perhaps more importantly, they don't capture more abstract properties of the code like maintainability.
AI works best in well maintained code, but unless care is taken, the AI will (today, anyway) make the code less well maintained as it goes.
If the AI is allowed to introduce mess into the codebase faster than its ability to deal with the mess increases, the codebase will eventually run into a problem that it's hard to recover from.