[redskyluan]

Based on your use cases. I thought it's not hard to push the window to 32K or even 100k if we change the position embedding