LLMs perform lossy compression on knowledge, so they're not a good replacement for indexing and search. If anything, you want to have the latter, and expose it to LLM as a tool it can use, regardless of whether the LLM was trained/fine-tuned on that same content or not.