[gnramires]

Also, you should note the memory and capabilities required to reach a conclusion might be much greater than to show it's true. Showing a needle may be easy, finding it in the haystack very hard. In this sense the hope for explainability is expanded. But still, I guess the real world is really messy "the full explanation" may be too large -- like when you explain a human intuition, the "full explanation" might have been your entire brain, your entire set of experiences up to that point; yet we can give partial explanations that should be satisfactory

A have a hypothesis that inevitably, reasoning needs to 'funnel' through explicit, logical representations (like we do with mathematics, language, etc.) to occur effectively. Or at least (quasi-)formalization is an important element of reasoning. This formal subset can be communicated.