[meowmeowmoo]

Can you elaborate on this window you reference? Or can you list any terms that I can use to read more about this? I'm really interested in learning about the limitations. Is this just a resource constraint or an algorithmic constraint?

[jerf]

This should at least get you going on the topic, in that even if it's too much or too little it's a rich source of terms, and directly addresses the topic: https://www.gwern.net/notes/Attention

I also want to make clear that while this is fundamental to this particular technology, I'm not saying it's fundamental to all possible AI architectures. But it is pretty ingrained into how transformers work. I don't think it can just "evolve" past it, I think anything that "evolved" past it would be a fundamentally different architecture.