[ISL]

It is hard to believe that modifying input datasets won't modify the qualitative behavior of the outputs in some way.

This appears to be a modern variation of the https://en.wikipedia.org/wiki/Fictitious_entry / copy-trap behavior that mapmakers have made in the past.

[rustyconover]

I think most ML models aren’t very “lean”, meaning there is space in their weight layers for information isn’t directly attributable to predictive accuracy. That space is likely where this new “radioactive” like data is being “stored”/“remembered”.

The leanness could be increased during training by progressively trimming width/depth of weights, but I doubt if every model has this done.

[lmkg]

This is definitely true. In fact, this can be exploited to extract sensitive/private attributes about the training data from the learned models. This may become an issue for, e.g., AI in healthcare.

"Overlearning Reveals Sensitive Attributes": https://arxiv.org/abs/1905.11742