[gart12one]

I think the difference here is that 'passing all the tests' isn't the same as satisfying the contract that the consumer is paying for. Even the best test suites won't cover all of the workflows a user might take.

Being pedantic here, but the two scenarios mentioned have regulations as well with repercussions for those products not satisfying their base requirements, whereas a lot of software is the wild-west (I won't be prosecuted for my software not actually working, in most cases).

[theshrike79]

But what the customer is paying for is defined somewhere, right? Preferably in writing.

You can give that to an agent and have it confirm the application fulfills the spec. Just like a human would have to.

And if the specific test can’t be automated, why? Is it impossible to automate or just inconvenient?

Granted there are immaterial things that we can’t automatically test yet like how the application feels to use. “Snappy” or “smooth” can’t really be quantified with code.