Look into RETRO, which greatly reduces the model’s tendency to confabulate by teaching it to query a document database known to be truthful, and justify its answers with specific references: https://www.deepmind.com/publications/improving-language-mod...