Of course, why didn't anyone think of that ? I bet if someone started to ship software that has no errors they'll make a huge amount of money, especially from all the people that are security-minded !
There is currently only method to prove absence of errors, this method is not LLMs, it's formal verification methods. Currently only very little formal verification is used in software industry, static type checking.