[danaris]

If there's an actual neurological attention window that you're referring to, that's as ephemeral as you say, it is not analogous to the context window of an LLM.

When I'm working on a problem, I can explicitly apply knowledge I gained decades ago to that problem. Not just "oh, it's somehow encoded in the training data"; not "oh, it's all mixed in there somewhere"; I can call up the memories, understand how to apply that knowledge to the problem at hand, and do so.

Except in certain very superficial ways, and despite many misguided LLM proponents' strident insistence to the contrary, LLMs do not work much like human brains.

[orbital-decay]

Yes, that's my entire point and I'm well aware about LLM internals. Reread the comment above, I'm responding to a person who suggested to increase the context window of an LLM to make it less different, saying it will never work.

[danaris]

I suppose that's one read of themafia's comment.

My read of it was "this is why humans are fundamentally different from LLMs", not "this is how you fix LLMs to be like humans".

It's still hard to see how you can argue that humans' "context window", looked at in even a vaguely similar manner to that of LLMs, is tiny.