If the context size is unbounded, how does the time complexity scale with the size of the context, and what are the limiting factors that affect performance as the context size grows larger?
The time and space complexity of inference is constant wrt. context size. You will probably need more parameters to match the performance of a transformer though, so whether it scales better in practice is an open question.