This idea is a simplified version of Retrieval-Augmented Generation (RAG), and RAG has been studied in various research papers, such as the one available at https://arxiv.org/abs/2005.11401
My experience with RAG is that while it reduces the incidence of hallucinations* significantly (especially if you reduce the LLM temperature to zero at the same time), it doesn't eliminate them.
My startup has a product for lawyers that uses RAG to answer legal queries (https://lawlight.ai/). We have a disclaimer that "... (we) do not guarantee the accuracy of answers. You are responsible for reviewing the cited case law and drawing your own independent conclusions."
(This works within the specific context—lawyers are domain experts; and they are supposed to read through all cases they cite in court anyway.)
* I dislike the term "hallucinations." By definition LLMs hallucinate. It's just that much (or most) of the time, the hallucinations reflect reality.
My startup has a product for lawyers that uses RAG to answer legal queries (https://lawlight.ai/). We have a disclaimer that "... (we) do not guarantee the accuracy of answers. You are responsible for reviewing the cited case law and drawing your own independent conclusions."
(This works within the specific context—lawyers are domain experts; and they are supposed to read through all cases they cite in court anyway.)
* I dislike the term "hallucinations." By definition LLMs hallucinate. It's just that much (or most) of the time, the hallucinations reflect reality.