[variadix]

This effect requires identical models, i.e. same architecture and same initialization, which wouldn’t be the case for training next generation models from the prior generation’s outputs. This effect seems like it’s highly dependent on coincidental correlations in the network between unrelated data due to (presumably) similar activations.

[gwern]

It's an open question how far this will transfer. Given the local basin/optima approach, and the incestuous nature of AI outputs + training, it's entirely possible that you could start to see 'lineages' of AIs (often undeclared, eg based on abusing APIs for distillation, and maybe unknown even to the creating entity if people/AI inside it are lying or hustling) where there is a lot of acausal coordination going on due to this.

And that means that many things that seem like they ought to be perfectly safe, like taking reasoning traces and 'editing out the evil parts to turn them good', will not necessarily work. (Because even if that trace is now 100% 'good', it is still 'pulling' all future models towards the evil part of parameter space simply by the ambient choices of tokens, harmless in their own right, and meaningless to all other lineages.)

[thorum]

It implies that training on synthetic data will always shift the model’s behavior in unpredictable ways. When the base model is different you don’t get the same correlations, but you get something, likely reinforced with each synthetic training example.

The greater variance of real world data might avoid this effect.