[delusional]

I think he's talking about computational efficiency. If you're loading in 29k tokens and you're expecting to use those again, you wouldn't need to do the whole matrix multiplication song and dance again if you just kept the old buffers around for the next prompt.

[weird-eye-issue]

I don't think this can necessarily be optimized at least with how the models work right now