|
|
|
|
|
|
by hospitalhusband
1135 days ago
|
|
|
> In what sense does an LLM think one word at a time that doesn't also apply to a person typing at a keyboard? Because you already have the thought formed before you started typing. > When I read my brain presumably ingests one word at a time (not sure if it's one exactly, but it can't be much more than one) And these models ingest many vectors at once, up to the context length. Your brain is also recursive, and regularly goes backwards to rescan earlier words as necessary. Seems to me it's fundamentally inverted from how we operate, both input and output. |
|
|
Can you prove that GPT-4 doesn't? Clearly there is a sense in which thinks more than one word ahead, since as I mentioned above it would not otherwise be able to use 'a' vs 'an' correctly.
As far as I am aware, exactly to what extent these models have determined what tokens will be generated before they produce anything is an open question in mechanistic interpratability research. I would be very interested if you knew of some work that answers this question empirically.