[vivegi]

That may be true with the human providing the explanation. However, there is at least some hope of being able to reason with the explanation (about the adequacy or lack thereof).

With a hallucinating LLM, most bets are off.

However, if the technology progresses to a level where the LLM output works like a proof of a theorem, verifiable step-by-step, there is a chance we could trust those systems. We are not there yet, I think.