[astrange]

Facebook had a paper about "system 2" LLM attention, where they identified which parts of the input would be distracting for the LLM and just deleted them.

https://arxiv.org/abs/2311.11829