[tipsytoad]

Could there be any merit training this on a common-sense dataset such as Cyc?

https://www.lesswrong.com/tag/cyc

[ipsum2]

Probably not, most common facts (sandcat is a type of feline) are already known by transformers. Maybe some obscure ones.