[imtringued]

One of the big limitations in LLMs is that they only have a single context window. People throw things that probably shouldn't be mixed together into the same context and hope for the best (e.g. system prompt, RAG context, user input, LLM output).

This is basically no different from a Turing machine going from one tape to multiple tapes. While in theory it doesn't make the Turing machine more powerful, it saves a whole lot of book keeping operations that are necessary to work around the limitations of a single tape.

Another limitation is the inability to seek to positions by moving the head back and forth to rewrite old data in the context.