[ralusek]

If you want an LLM to have knowledge about something, the knowledge has to either exist in its weights or be provided to it in its context. Because context is expensive and limited, and models tend to get dumber the more their context is filled, there is usually more that you'd need to put into context than can reasonably fit in it in order for the model to answer questions about your data. So your options are basically

1.) stuff it into context

2.) figure out a way to determine what to put into context based off of what is being asked of the model (RAG)

3.) change the weights of the model to have knowledge of your data baked into it (fine tuning)

[probe]

Do you think 3 is better than 1 & 2 as context gets larger? I think for smaller data sets its mostly fine no. It's an interesting bet by TM. End state does seem some form of continual learning (model weights update like dreaming)