Can you say more? This sounds like it could be more useful than trying to build a knowledge base (say qmd) and then get the models to search it for the right context each time.
This is in fact what we do (with higher order abstractions now built on top of this). This builds self evolving interactive knowledge base and puts it into a QMD searchable index. The indexer is already open source: https://github.com/jibs/duffel