[choeger]

Did anyone ever attempt a recursive architecture?

So you take the first window or logical separatation (chapter, paragraph) and let the model summarize it into one or two sentences. Then you repeat that with the next window (and that derived sentence as context) and create a new logical separatation out out of a fixed number of sentences. Rinse and repeat until the result fits into your window.

I have a hunch that this is somewhat how the brain works when reading.

[brrrrrm]

Hierarchical attention gets you close https://arxiv.org/abs/2210.05529