[vidarh]

Look up split brain experiments [1]. Basically, in patients where the corpus callosum is severed to some degree, the two halves of the brain have limited communication. Since the two halves control different parts of the sensory system, you can provide information selectively to one or the other half of the brain, and ask that part of the brain to make choices. If you then provide the other half of the brain the wrong information, and ask it to reason about why it made a choice, the other brain will happily pull a ChatGPT and "hallucinate" a reason for a choice neither that half nor the other half of the brain ever made.

While that does not prove that we never apply reasoning ahead of time, it is a pretty compelling indication that we can not trust that reasoning we give isn't a post-rationalisation rather than an indication of our actual reasoning, if any.

[1] https://en.wikipedia.org/wiki/Split-brain