[krainboltgreene]

For the same reason you can't reduce a human to simply a bag of atoms and expect to understand the person.

[naasking]

But humans are a specific type of a bag of atoms, and humans do (mostly) understand what they say and do, so that's not a legitimate argument against the reducibility of "understanding" to a such a bag of atoms (or specific kind of next token prediction for LLMs).