[agucova]

Note that:

> ML cannot conceptualize of things in the abstract like people can

And:

> They cannot offer reasons, a train of thought like a person

Are very different claims! The first one just seems wrong: LLMs require abstraction to work, and early work in interpretability suggests they build rich world models during training (i.e. see https://thegradient.pub/othello/).

What is true is that often those models aren’t very legible, and it would seem current LLMs are incapable of introspection, and so can’t make those models more transparent.

The second one is a tricky one: you can often get it by explicitly prompting for a chain of thought, but it’s true current LLMs don’t seem great at this yet. The big jump in this capability when going from GPT 3.5 to GPT 4 makes me thing that this is just a limitation that will be overcome relatively soon.