[hodgehog11]

As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate?

By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.

[root_axis]

> They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs.

How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?

[rerdavies]

The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.

[miyoji]

> (consciousness being a cognitive capacity)

I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all.

So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.

[morpheos137]

allow me to resolve the confusion: ai is a model of language. language is a model of human cognitive state. to the extent language maps to cognitive state accurately and ai processes language fluently ai models understanding. whether this is understanding is an irrelevant metaphysical question to me.

[miyoji]

> language is a model of human cognitive state.

This is false. Cognitive states do not require language and language is an insufficient model of any cognitive state.

The follow-ons for the rest of your so-called resolution should be clear.

[morpheos137]

'This is false,' that is an assertion not a proof, and the assertion rests on some kind of asbolutizing fallacy where if x =! y then y =! y(x). I agree cognitive states do not require language. However to say language does not model cognitive states is absurd, from where else would language derive meaning? No I am not saying it models the specific mechanics of neurons. Rather I am saying language maps to the abstract meaning of cognitive states. So the word red absolutely does correspond to some class of brain states common to red. What else could possibly allow cognition to interface with the physical world. Note 'models,' does not need to mean 'injectively, transitively' corresponds. the fact language does not cover all cognitive states in full in no way invalidates language as a model of cognitive state, in the same way a map is a model of territory without being the territory.

[hodgehog11]

"LLMs could not possibly be good models of some cognitive capacity because they are just a bunch of numbers guessing the next word. They have no linguistic module, so they cannot exhibit cognition". That's the argument. It's pretty clearly stated.

Look, this isn't necessarily directed at you, but I've been a researcher into the theory of deep learning for many years now. I've seen all the phases, heard all the criticism, had to constantly argue against this. Gary Marcus was one of the loudest voices of this argument, but every would-be philosopher came out of the woodwork to explain why LLMs are no more than stochastic parrots because of their design. Geoffrey Hinton famously had to debunk these arguments multiple times.

And now that LLMs start to clearly exhibit intelligent behavior and can be somewhat reliable, now "nobody ever thought that LLMs could not possibly be good models of some cognitive capacity because of next-token predictions, or linear algebra, etc."? No, that's not okay.