[jahooma]

We actually don't use RAG! It's not that good as you say.

We build a description of the codebase including the file tree and parsed function names and class names, and then just ask Haiku which files are relevant!

This works much better and doesn't require slowly creating an index. You can just run Codebuff in any directory and it works.

[michaelmior]

It sounds like it's arguably still a form of RAG, just where the retrieval is very different. I'm not saying that to knock your approach, just saying that it sounds like it's still the case where you're retrieving some context and then using that context to augment further generation. (I get that's definitely not what people think of when you say RAG though.)

[brandonchen]

Genuine question: at what point does the term RAG lose its meaning? Seems like LLMs work best when they have the right context, and that context must be pulled from somewhere for the LLM. But if that's RAG, then what isn't? Do you have a take on this? Been struggling to frame all this in my head, so would love some insight.

[parsimo2010]

RAG is a search step in an attempt to put relevant context into a prompt before performing inference. You are “augmenting” the prompt by “retrieving” information from a data set before giving it to an LLM to “generate” a response. The data set may be the internet, or a code base, or text files. The typical examples online uses an embedding model and a vector database for the search step, but doing a web query before inference is also RAG. Perplexity.ai is a RAG (but fairly good quality). I would argue that Codebuff’s directory tree search to find relevant files is a search step. It’s not the same as a similarity search on vector embeddings, and it’s not PageRank, but it is a search step.

Things that aren’t RAG, but are also ways to get a LLM to “know” things that it didn’t know prior:

1. Fine-tuning with your custom training data, since it modifies the model weights instead of adding context. 2. LoRA with your custom training data, since it adds a few layers on top of a foundation model. 3. Stuffing all your context into the prompt, since there is no search step being performed.

[brandonchen]

Gotcha – so broadly encompasses how we give external context to the LLM. Appreciate the extra note about vector databases, that's where I've heard this term used most, but I'm glad to know it extends beyond that. Thanks for explaining!

[petesergeant]

Not RAG: asking the LLM to generate using its internal weights only

RAG: providing the LLM with contextual data you’ve pulled from outside its weights that you believe relate to a query

[brandonchen]

Nice, super simple. We're definitely fitting into this definition of RAG then!

[michaelmior]

I think parsimo2010 gave a good definition. If you're pulling context from somewhere using some search process to include as input to the LLM, I would call that RAG.

So I would not consider something like using a system prompt (which does add context, but does not involve search) would not be RAG. Also, using an LLM to generate search terms before returning query results would not be RAG because the output of the search is not input to the LLM.

I would also probably not categorize a system similar to Codebuff that just adds the entire repository as context to be RAG since there's not really a search process involved. I could see that being a bit of a grey area though.

[petesergeant]

> We build a description of the codebase including the file tree and parsed function names and class names

This sounds like RAG and also that you’re building an index? Did you just mean that you’re not using vector search over embeddings for the retrieval part, or have I missed something fundamental here?

[jahooma]

Ah yeah, that's what I mean! I thought RAG is synonymous with this vector search approach.

Either way, we do the search step a little different and it works well.

[cratermoon]

Any kind of search prior for content to provide as context to the LLM prompt is RAG. The goal is to leverage traditional information retrieval as a source of context. https://cloud.google.com/use-cases/retrieval-augmented-gener...

I'm currently working on a demonstration/POC system using my ElasticSearch as my content source, generating embeddings from that content, and passing them to my local LLM.

[petesergeant]

It would be cool to be talking to other people about the RAG systems they’re building. I’m working in a silo at the moment, and pretty sure that I’m reinventing a lot of techniques

[petesergeant]

I didn't mean to be down on it, and I'm really glad it's working well! If you start to reach the limits of what you can achieve with your current approach, there are lots of cute tricks you can steal from RAG, eg nothing stopping you doing a fuzzy keyword search for interesting-looking identifiers on larger codebases rather than giving the LLM the whole thing in-prompt, for example