[kromem]

Depth isn't always the right approach though.

Personally, I'm much more excited at the idea of pairing RAG with a 1M token context window to have enormous effective breadth in a prompt.

For example, you could have RAG grab the relevant parts of every single academic paper related to a given line of inquiry and provide it into the context to effectively perform a live meta-analyses with accurate citation capabilities.

[whakim]

I really don’t think the issue with RAG is the size of the context window. In your example, the issue is selecting which papers to use, because most RAG implementations rely on naive semantic search. If the answer isn’t to be found in text that is similar to the user’s query (or the paper containing that text) then you’re out of luck. There’s also the complete lack of contextual information - you can pass 100 papers to an LLM, but the LLM has no concept of the relationship between those papers, how they interact with each other and the literature more broadly (beyond what’s stated in the text), etc. etc.