Taking this to the extreme, should providers of other services not censor their products? This includes KYC/AML for financial institutions and section 230 like provisions for social media. It seems like society deems those to be beneficial, why not similar for hosted AI products?
Note that self-hosted models do not have these restrictions.
By "society" you mean "banks on behalf of law enforcement as a hack to make their job easier". I think most people hate the oppressive red tape and growing troves of ready-to-leak credentials companies are increasingly incentivized to hoover up from users. While you're at it, "society deems DRM and NSA surveillance to be beneficial".
The banks are liable if they don’t police their users, just like you would be liable if you in turn gave your credentials out to someone else.
Society apparently has deemed those beneficial at large as there is not a critical mass of opposition to get representatives to push for changes. In this case I’m equating special interest groups and PAC’s with money to spend to influence as also beneficial to society as they have been allowed to prosper.
There are AI models, which are free to download and run locally, that are uncensored. That is the reality today.
Then there are platforms that host models they have developed. Is it not ok for those proprietary hosting platforms to apply guardrails to what their models respond with?
Note that self-hosted models do not have these restrictions.