I think what's interesting is when these LLM return responses that we agree with, it's nothing special. It's only when they respond with what humans deem "uhhhh" that we point and discuss.
I think it's even more interesting that these models actually return meaningless vectors that we then translate into text.
It makes you think a lot about how human talk. We can't just be probabilistically stringing together word tokens, we think in terms of meaning, right? Maybe?
> We can't just be probabilistically stringing together word tokens, we think in terms of meaning, right?
We are probabalistically stringing together muscle movements that generate language as sound. That's not really controversial, otherwise we would call it magic. However, the complexity of our probabalistic word machine is far greater, in terms of both richness of inputs, motivation, and dimensionality.
There is a huge universe of inputs, aka training data, that feeds into us, far more than a digital text based LLM. From that we generated the training data for the LLM. That data is just a sliver of the human experience.
> The universe contained exactly 0 words until humans created them,
Human words are one fork of the sound wave based communication systems that many animals on earth use. There was no distinct moment when we went from 0 to 1 words. There was no "first person to speak". We didn't make language. It emerged over time due to evolutionary pressures.
It makes you think a lot about how human talk. We can't just be probabilistically stringing together word tokens, we think in terms of meaning, right? Maybe?