[svara]

It has learnt all sorts of invariances, almost certainly also that.

I've gotten some very weird results with 4o on images, it seems entirely possible to me that it would go off the rails if the image wasn't in the training data.

For this specific case, it's really not easy to test at all.

[sillysaurusx]

An invariant isn’t the same thing as a purple giraffe. One is an image manipulation applied at training time to make the classifier robust against transformations. The other is a thing that might someday exist in nature. (The most straightforward way is to dump a barrel of wine over the giraffe and take a photo.)

[svara]

You're thinking simple image augmentations. These nets learn much more complex invariants. Basically to isolate concepts from irrelevant context. The point is you can't remove that image from the training data (not practically) and the experiment is pointless if it's in there.

[sillysaurusx]

Sure you can. Have it generate a new photo. Or dump a barrel of wine over the giraffe.