[seri4l]

There are some experiments that suggest that human actions are not the result of conscious thought. Maybe you've heard about Michael Gazzaniga's experiments on split brain subjects. The most radical interpretation of the results is that our conscious thought merely constructs a narrative about actions it has no direct control of.

In other words, our brains act like a bunch of interconnected neural networks and one of them, the language processing part, tries to make sense of what all the others are doing, serving the only purpose of communicating with others.

[simonh]

I wasn’t aware of that research, thanks, but I do agree most cognition is subconscious. When talking rapidly, the words seem to just spill out from nowhere. Ideas simply spring to mind. We have no direct awareness if the cognitive processes that generate them.

We can reason consciously step by step, but it’s an incredibly slow process. It seems likely that consciousness has something to do with attention management and broadcast communication across brain regions and functions.

[TeMPOraL]

Which is part of the reason why I think LLMs are a big deal. LLMs talk like the inner voice, and their failure modes are strikingly similar to the failure modes of "gut instinct" / first reactions and "stream of thoughts". I don't think this is a coincidence - I think we may have stumbled on the main trick that makes biological brains work. I don't mean language here, but rather the use of absurdly high-dimensional latent space for representing relatedness.

Now, if the above is anywhere close to the truth, then taking it together with the research you mention suggests that LLMs aren't simulating or parroting abstract thinking and understanding - they're actually doing it, the same way we do. They just lack an evaluator/censor layer of conscious experience.