A straightforward point that comes to mind regards reusing the output of GPT-n as part of the training input to GPT-n+1: Andrej Karpathy recently tweeted about it in terms of pollution, which has wonderful connotations: https://twitter.com/karpathy/status/1284660899198820352
So far most of this pollution is effectively tagged through accompanying mentions of "GPT", but filtering that from future training data would mean GPT can never learn about itself.