[Tarean]

The transformer model in GPT-3 has a short context window and no recurrence. Without some significant architecture changes that is a fundamental limit on the problems GPT-3 can solve.

[lucidrains]

https://arxiv.org/abs/1807.03819