[rustyconover]

I think most ML models aren’t very “lean”, meaning there is space in their weight layers for information isn’t directly attributable to predictive accuracy. That space is likely where this new “radioactive” like data is being “stored”/“remembered”.

The leanness could be increased during training by progressively trimming width/depth of weights, but I doubt if every model has this done.

[lmkg]

This is definitely true. In fact, this can be exploited to extract sensitive/private attributes about the training data from the learned models. This may become an issue for, e.g., AI in healthcare.

"Overlearning Reveals Sensitive Attributes": https://arxiv.org/abs/1905.11742