[mrbungie]

I would guess correctly aligned and/or finely filtered synthetic data coming from LLMs may be good.

Mode colapse theories (and simplified models used as proof of existence of said problem) assume affected LLMs are going to be trained with poor quality LLM-generated batches of text from the internet (i.e. reddit or other social networks).