[mattnewton]

I recently wrote a short anecdote in a similar vein- in my testing, “agentic” retrieval where you simply pass an annotated list of files to an LLM and ask it which ones it wants to look at is probably better than traditional RAG for small datasets (few hundreds of docs).

I found it was both much simpler and more accurate at the cost of marginally more time and tokens, compared to RAG on embedded chunks with a vector store.

Shameless plug- https://www.matthewnewton.com/blog/replacing-rag

[jarirajari]

I was daunted by the amount of components that goes into a basic RAG. I also ended up starting with "agentic retrieval", which I invented. And it seems that many others have invented same/similar thing. It is just so much easier to start with something simple and improve later on.

[mattnewton]

Exactly, If it’s dumb and it works it isn’t dumb. (Just have to measure that it does work).

I think this is one of those cases where doing the simplest possible thing is suprisingly good and getting better as LLMs get better. I thought it would be a placeholder with all kinds of problems I would have to hurry to replace but it was surprisingly hard to beat on our most important metric of accuracy. And accuracy got better for “free” when the underlying model got better.