[GaggiX]

>In pre-training, we filter out(opens in a new window) information that we do not want our models to learn from or output, such as hate speech, adult content, sites that primarily aggregate personal information, and spam.

Great so now the model would be unable to recognize this type of content, do not use it for moderation.

[93po]

I think this is a strong conclusion to jump to. Maybe it's better at spotting content that needs to be moderated because it stands out more from what it's been trained on?

[GaggiX]

This is not really how these models work, if the sample is out of distribution then it would usually perform worse on the task assigned.