Why is this illegal btw? I mean whats stopping an AI company from releasing a proper NSFW model? I hope it doesn't happen but I want to know what prevents them from doing it now.
in some jurisdictions generating a swastika or a hammer and sickle is illegal.
that said, I'm sure you can imagine that the really illegal, truly, positively sickening and immoral stuff is children-adjacent and you can be 100% sure there are sociopaths doing training runs for the broken people who'll buy the weights.
Not in the United States, but it is illegal in some jurisdictions.
Additionally, the entire "payment processors leaning on Steam" thing shows that it might be very difficult to monetize a model that's known for generating extremely controversial content. Without monetization, it would be hard for any company to support the training (and potential release) of an unshackled enterprise-grade model.
Most of Europe doesn't really have free speech, frankly most of the world doesn't. Privileges like making mspaint drawings of nearly whatever you want is pretty uniquely American.
that said, I'm sure you can imagine that the really illegal, truly, positively sickening and immoral stuff is children-adjacent and you can be 100% sure there are sociopaths doing training runs for the broken people who'll buy the weights.