[minimaxir]

That isn't how LLM training has worked for some time. There's a reason the LLM boom didn't take off until training was separated into pretraining (training on all data) and posttraining (RLHF to make the output actually aligned).

It's also why model collapse is not a thing despite everyone wanting it to be.