[Eridrus]

Is ALiBi still the sota for this setting, or have there been advances beyond this in the last 8 months? I know there has been a lot of interest in longer context lengths recently.

[ipsum2]

xpos is SoTA right now: https://arxiv.org/pdf/2212.10554.pdf

[Eridrus]

Thanks!