I agree that all of this is madness, but to be fair, 100% of computer programs have bugs, so I don't think this is any more damning than systematizing all of humanity's processes with computers.
It's about the likelihood and probability distribution of the kinds of bugs.
An Excel sheet made by NASA as part of their official space operations, to that rigor will probably have 0 problems ever. But compared to typical software, just consider this: Excel sheets do not have a culture of any kind of testing. If the output looks plausible, move on. Garbage in, garbage processing, garbage out.
AI slop has a similar but not exactly the same problem: It's highly, highly likely a human "reviewing" the generated code will just nod their head and not think critically/independently, just accepting all things that are superficially plausible. That may be fine for your webdev CRUD but I sure hope those people are never given credentials that can access anything critical.
An Excel sheet made by NASA as part of their official space operations, to that rigor will probably have 0 problems ever. But compared to typical software, just consider this: Excel sheets do not have a culture of any kind of testing. If the output looks plausible, move on. Garbage in, garbage processing, garbage out.
AI slop has a similar but not exactly the same problem: It's highly, highly likely a human "reviewing" the generated code will just nod their head and not think critically/independently, just accepting all things that are superficially plausible. That may be fine for your webdev CRUD but I sure hope those people are never given credentials that can access anything critical.