[sjy]

> If the insured sues the insurance company, the company can bring their engineers as witnesses to explain how the denial logic has been coded.

Sometimes I wonder how useful this option is in practice. The British Post Office scandal [1] is a rare example where engineers actually were called as witnesses to explain the logic of a complex enterprise software system with legally significant consequences. The cost of just getting to the 313-page judgment addressing the technical issues [2] was astronomical. Some of the hundreds of wrongfully convicted small business people had gone to trial and tried, but failed, to discredit the prosecution's expert evidence that the software was reliable. If the decisions had come out of an inscrutable LLM, it might have saved a lot of trouble.

[1]: https://en.wikipedia.org/wiki/British_Post_Office_scandal

[2]: https://www.judiciary.uk/wp-content/uploads/2019/12/bates-v-...

[vivegi]

That may be true with the human providing the explanation. However, there is at least some hope of being able to reason with the explanation (about the adequacy or lack thereof).

With a hallucinating LLM, most bets are off.

However, if the technology progresses to a level where the LLM output works like a proof of a theorem, verifiable step-by-step, there is a chance we could trust those systems. We are not there yet, I think.