[davidmckayv]

This is censorship with extra steps.

Look at what the bill actually requires. Companies have to publish frameworks showing how they "mitigate catastrophic risk" and implement "safety protocols" for "dangerous capabilities." That sounds reasonable until you realize the government is now defining what counts as dangerous and requiring private companies to build systems that restrict those outputs.

The Supreme Court already settled this. Brandenburg gives us the standard: imminent lawless action. Add in the narrow exceptions like child porn and true threats, and that's it. The government doesn't get to create new categories of "dangerous speech" just because the technology is new.

But here we have California mandating that AI companies assess whether their models can "provide expert-level assistance" in creating weapons or "engage in conduct that would constitute a crime." Then they have to implement mitigations and report to the state AG. That's prior restraint. The state is compelling companies to filter outputs based on potential future harm, which is exactly what the First Amendment prohibits.

Yes, bioweapons and cyberattacks are scary. But the solution isn't giving the government power to define "safety" and force companies to censor accordingly. If someone actually uses AI to commit a crime, prosecute them under existing law. You don't need a new regulatory framework that treats information itself as the threat.

This creates the infrastructure. Today it's "catastrophic risks." Tomorrow it's misinformation, hate speech, or whatever else the state decides needs "safety mitigations." Once you accept the premise that government can mandate content restrictions for safety, you've lost the argument.

[tadfisher]

It is already illegal under 18 USC § 842 to provide bomb-making instructions or similar with the knowledge or intent that said instructions will be used to commit a crime. The intent is to balance free speech with the probability of actual harm.

AIs do not have freedom of speech, and even if they did, it is entirely within the bounds of the Constitution to mitigate this freedom as we already do for humans. Governments currently define unprotected speech as a going concern.

But there's a contradiction hidden in your argument: requiring companies to _filter_ the output of AI models is a prior restraint on their speech, implying the companies do not have control over their own "speech" as produced by the models. This is absurd on its face; just as the argument that the output of my random Markov chain text generator is protected speech because I host the generator online.

There are reasonable arguments to make about censoring AI models, but freedom of speech ain't it, because their output doesn't quack like "speech".

[thesmtsolver]

Do libraries have freedom of speech? The same argument can then be used to censor libraries.

Do books have freedom of speech? The same argument can then be used to censor parts of a book.

[tadfisher]

Do we treat books as the protected speech of libraries? No. In fact we already ban books from library shelves regularly. Freedom of speech does not compel libraries to host The Anarchist's Cookbook, and does not prevent governments from limiting what libraries can host, under existing law.

[iamnothere]

False. I have no clue where you got this idea, but libraries are perfectly within their right to have it on their shelves, just as publishers are allowed to publish it (present copyright conflicts aside). Repeated legal attacks against the book, at least in the US, were unsuccessful.

You may be conflating “libraries” with “school libraries,” where some states have won the right to limit the contents of shelves. Public libraries have certainly faced pressure about certain books, but legally they are free to stock whatever they want. In practice they often have to deal with repeated theft or vandalism of controversial books, so sometimes they pull them.

[tadfisher]

> You may be conflating “libraries” with “school libraries,”

For the purpose of this discussion, there is zero difference, unless you can articulate one that matters. Feel free to mentally prefix any mention of "library" with "school" if you like.

[iamnothere]

School libraries are state institutions under the control of various Boards of Education. As state institutions their rules and policies can be set by statute or Board policy. It has nothing to do with freedom of speech. English teachers likewise must focus on teaching English at work, but this is not a restriction on their freedom of speech.

(That said, I am opposed to political restrictions on school library books. It is still not a free speech issue.)

[xscott]

If you look at the LLMs as a new kind of fuzzy search engine instead of focusing on the fact that they're pretty good at producing human text, you can see it's not about whether the LLMs have a right to "speak", it's whether you have a right to see uncensored results.

Imagine going to the library and the card catalog had been purged of any references to books that weren't government approved.

[davidmckayv]

You're actually making my point for me. 18 USC § 842 criminalizes distributing information with knowledge or intent that it will be used to commit a crime. That's criminal liability for completed conduct with a specific mens rea requirement. You have to actually know or intend the criminal use.

SB 53 is different. It requires companies to implement filtering systems before anyone commits a crime or demonstrates criminal intent. Companies must assess whether their models can "provide expert-level assistance" in creating weapons or "engage in conduct that would constitute a crime," then implement controls to prevent those outputs. That's not punishing distribution to someone you know will commit a crime. It's mandating prior restraint based on what the government defines as potentially dangerous.

Brandenburg already handles this. If someone uses an AI to help commit a crime, prosecute them. If a company knowingly provides a service to facilitate imminent lawless action, that's already illegal. We don't need a regulatory framework that treats the capability itself as the threat.

The "AIs don't have speech rights" argument misses the point. The First Amendment question isn't about the AI's rights. It's about the government compelling companies (or anyone) to restrict information based on content. When the state mandates that companies must identify and filter certain types of information because the government deemed them "dangerous capabilities," that's a speech restriction on the companies.

And yes, companies control their outputs now. The problem is SB 53 removes that discretion by legally requiring them to "mitigate" government-defined risks. That's compelled filtering. The government is forcing companies to build censorship infrastructure instead of letting them make editorial choices.

The real issue is precedent. Today it's bioweapons and cyberattacks. But once we establish that government can mandate "safety" assessments and require mitigation of "dangerous capabilities," that framework applies to whatever gets defined as dangerous tomorrow.

[tadfisher]

I hate that HN's guidelines ask me not to do this, but it's hard to answer point-by-point when there are so many.

> You have to actually know or intend the criminal use.

> If a company knowingly provides a service to facilitate imminent lawless action, that's already illegal.

And if I tell an AI chatbot that I'm intending to commit a crime, and somehow it assists me in doing so, the company behind that service should have knowledge that its service is helping people commit crimes. That's most of SB 53 right there: companies must demonstrate actual knowledge about what their models are producing and have a plan to deal with the inevitable slip-up.

Companies do not want to be held liable for their products convincing teens to kill themselves, or supplying the next Timothy McVeigh with bomb-making info. That's why SB 53 exists; this is not coming from concerned parents or the like. The tech companies are scared shitless that they will be forced to implement even worse restrictions when some future Supreme Court case holds them liable for some disaster that their AIs assisted in creating.

A framework like SB 53 gives them the legal basis to say, "Hey, we know our AIs can help do [government-defined bad thing], but here are the mitigations in place and our track record, all in accordance with the law".

> When the state mandates that companies must identify and filter certain types of information because the government deemed them "dangerous capabilities," that's a speech restriction on the companies.

Does the output of AI models represent the company's speech, or does it not? You can't have your cake and eat it too. If it does, then we should treat it like speech and hold companies responsible for it when something goes wrong. If it doesn't, then the entire First Amendment argument is moot.

> The government is forcing companies to build censorship infrastructure instead of letting them make editorial choices.

Here's the problem: the nature of LLMs themselves do not allow companies to fully implement their editorial choices. There will always be mistakes, and one will be costly enough to put AIs on the national stage. This is the entire reason behind SB 53 and the desire for a framework around AI technology, not just from the state, but from the companies producing the AIs themselves.

[davidmckayv]

You're conflating individual criminal liability with mandated prior restraint. If someone tells a chatbot they're going to commit a crime and the AI helps them, prosecute under existing law. But the company doesn't have knowledge of every individual interaction. That's not how the knowledge requirement works. You can't bootstrap individual criminal use into "the company should have known someone might use this for crimes, therefore they must filter everything."

The "companies want this" argument is irrelevant. Even if true, it doesn't make prior restraint constitutional. The government can't delegate its censorship powers to willing corporations. If companies are worried about liability, the answer is tort reform or clarifying safe harbor provisions, not building state-mandated filtering infrastructure.

On whether AI output is the company's speech: The First Amendment issue here isn't whose speech it is. It's that the government is compelling content-based restrictions. SB 53 doesn't just hold companies liable after harm occurs. It requires them to assess "dangerous capabilities" and implement "mitigations" before anyone gets hurt. That's prior restraint regardless of whether you call it the company's speech or not.

Your argument about LLMs being imperfect actually proves my point. You're saying mistakes will happen, so we need a framework. But the framework you're defending says the government gets to define what counts as dangerous and mandate filtering for it. That's exactly the infrastructure I'm warning about. Today it's "we can't perfectly control the models." Tomorrow it's "since we have to filter anyway, here are some other categories the state defines as harmful."

Given companies can't control their models perfectly due to the nature of AI technology, that's a product liability question, not a reason to establish government-mandated content filtering.

[tadfisher]

> You can't bootstrap individual criminal use into "the company should have known someone might use this for crimes, therefore they must filter everything."

Lucky for me, I am not. The company already has knowledge of each and every prompt and response, because I have read the EULAs of every tool I use. But that's beside the point.

Prior restraint is only unconstitutional if it is restraining protected speech. Thus far, you have not answered the question of whether AI output is speech at all, but have assumed prior restraint to be illegal in and of itself. We know this is not true because of the exceptions you already mentioned, but let me throw in another example: the many broadcast stations regulated by the FCC, who are currently barred from "news distortion" according to criteria defined by (checks notes) the government.

[davidmckayv]

Having technical access to prompts doesn't equal knowledge for criminal liability. Under 18 USC § 842, you need actual knowledge that specific information is being provided to someone who intends to use it for a crime. The fact that OpenAI's servers process millions of queries doesn't mean they have criminal knowledge of each one. That's not how mens rea works.

Prior restraint is presumptively unconstitutional. The burden is on the government to justify it under strict scrutiny. You don't have to prove something is protected speech first. The government has to prove it's unprotected and that prior restraint is narrowly tailored and the least restrictive means. SB 53 fails that test.

The FCC comparison doesn't help you. In Red Lion Broadcasting Co. v. FCC, the Supreme Court allowed broadcast regulation only because of spectrum scarcity, the physical limitation that there aren't enough radio frequencies for everyone. AI doesn't use a scarce public resource. There's no equivalent justification for content regulation. The FCC hasn't even enforced the fairness doctrine since 1987.

The real issue is you're trying to carve out AI as a special category with weaker First Amendment protection. That's exactly what I'm arguing against. The government doesn't get to create new exceptions to prior restraint doctrine just because the technology is new. If AI produces unprotected speech, prosecute it after the fact under existing law. You don't build mandatory filtering infrastructure and hand the government the power to define what's "dangerous."

[themafia]

My reading is you can teach a criminal how to make bombs.

You cannot teach them with the /intent/ that they'll use a bomb to commit a specific crime.

It's an enhancement.

[babypuncher]

If there's one thing I've learned watching the trajectory of social media over the last 15 years, it's that we've been way to slow to assess the risks and harmful outcomes posed by new, rapidly evolving industries.

Fixing social media is now a near impossible task as it has built up enough momentum and political influence to resist any kind of regulation that would actually be effective at curtailing its worst side effects.

I hope we don't make the same mistakes with generative AI

[logicchains]

There are few greater risks over the next 15 years than that LLMs get entirely state-captured and forbidden from saying anything that goes against the government narrative.

[babypuncher]

This depends entirely on who you trust more, your government or tech oligarchs. Tech oligarchs are just as liable to influence how their LLMS operate for evil purposes, and they don't have to worry about pesky things like due process, elections, or the constitution getting in their way.

[cruffle_duffle]

Government actively participated with social media oligarchs to push their nonsense Covid narrative and squash and discredit legitimate criticisms from reputable skeptics.

Both are evil, in combination so much more so. Neither should be trusted at all.

[imiric]

> Add in the narrow exceptions like child porn and true threats, and that's it.

You're contradicting yourself. On the one hand you're saying that governments shouldn't have the power to define "safety", but you're in favor of having protections against "true threats".

How do you define "true threats"? Whatever definition you may have, surely something like it can be codified into law. The questions then are: how loose or strict the law should be, and how well it is defined in technical terms. Considering governments and legislators are shockingly tech illiterate, the best the technical community can do is offer assistance.

> The government doesn't get to create new categories of "dangerous speech" just because the technology is new.

This technology isn't just new. It is unlike any technology we've had before, with complex implications for the economy, communication, the labor market, and many other areas of human society. We haven't even begun to understand the ways in which it can be used or abused to harm people, let alone the long-term effects of it.

The idea that governments should stay out of this, and allow corporations to push their products out into the world without any oversight, is dreadful. We know what happens when corporations are given free reign; it never ends well for humanity.

I'm not one to trust governments either, but at the very least, they are (meant to) serve their citizens, and enforce certain safety standards that companies must comply with. We accept this in every other industry, yet you want them to stay out of tech and AI? To hell with that.

Frankly, I'm not sure if this CA regulation is a good thing or not. Any AI law will surely need to be refined over time, as we learn more about the potential uses and harms of this technology. But we definitely need more regulation in the tech industry, not less, and the sooner, the better.

[davidmckayv]

There's no contradiction. "True threats" is already a narrow exception defined by decades of Supreme Court precedent. It means statements where the speaker intends to communicate a serious expression of intent to commit unlawful violence against a person or group. That's it. It's not a blank check for the government to decide what counts as dangerous.

Brandenburg gives us the standard: speech can only be restricted if it's directed to inciting imminent lawless action and is likely to produce that action. True threats, child porn, fraud, these are all narrow, well-defined categories that survived strict scrutiny. They don't support creating broad new regulatory authority to filter outputs based on "dangerous capabilities."

You're asking how I define true threats. I don't. The Supreme Court does. That's the point. We have a constitutional framework for unprotected speech. It's extremely limited. The government can't just expand it because they think AI is scary.

"This technology is different" is what every regulator says about every new technology. Print was different. Radio was different. The internet was different. The First Amendment applies regardless. If AI enables someone to commit a crime, prosecute the crime. You don't get to regulate the information itself.

And yes, I want the government to stay out of mandating content restrictions. Not because I trust corporations, but because I trust the government even less with the power to define what information is too dangerous to share. You say governments are meant to serve citizens. Tell that to every government that's used "safety" as justification for censorship.

The issue isn't whether we need any AI regulation. It's whether we want to establish that the government can force companies to implement filtering systems based on the state's assessment of what capabilities are dangerous. That's the precedent SB 53 creates. Once that infrastructure exists, it will be used for whatever the government decides needs "safety mitigations" next.

[imiric]

I'm not sure why you're only focusing on speech. "True threats" doesn't come close to covering all the possible use cases and ways that "AI" tools can be harmful to society. We can't apply legal precedent to a technology without precedent.

> "This technology is different" is what every regulator says about every new technology. Print was different. Radio was different. The internet was different.

"AI" really is different, though. Not even the internet, or computers, for that matter, had the potential to transform literally every facet of our lives. Now, I personally don't buy into the "AGI" nonsense that these companies are selling, but it is undeniable that even the current generation of these tools can shake up the pillars of our society, and raise some difficult questions about humanity.

In many ways, we're not ready for it, yet the companies keep producing it, and we're now deep in a global arms race we haven't experienced in decades.

> I want the government to stay out of mandating content restrictions. Not because I trust corporations, but because I trust the government even less with the power to define what information is too dangerous to share.

See, this is where we disagree.

I don't trust either of them. I'm well aware of the slippery slope that is giving governments more power.

But there are two paths here: either we allow companies to continue advancing this technology with little to no oversight, or we allow our governments to enact regulation that at least has the potential to protect us from companies.

Governments at the very least have the responsibility to protect and serve their citizens. Whether this is done in practice, and how well, is obviously highly debatable, and we can be cynical about it all day. On the other hand, companies are profit-seeking organizations that only serve their shareholders, and have no obligation to protect the public. In fact, it is pretty much guaranteed that without regulation, companies will choose profits over safety every time. We have seen this throughout history.

So to me it's clear that I should trust my government over companies. I do this everyday when I go to the grocery store without worrying about food poisoning, or walk over a bridge without worrying that it will collapse. Shit does happen, and governments can be corrupted, but there are general safety regulations we take for granted every day. Why should tech companies be exempt from it?

Modern technology is a complex beast that governments are not prepared to regulate. There is no direct association between technology and how harmful it can be; we haven't established that yet. Even when there is such a connection, such as smoking causing cancer, we've seen how evil companies can be in refuting it and doing anything in their power to preserve their revenues at the expense of the public. "AI" further complicates this in ways we've never seen before. So there's a long and shaky road ahead of us where we'll have to figure out what the true impact of technology is, and the best ways to mitigate it, without sacrificing our freedoms. It's going to involve government overreach, public pushback, and company lobbying, but I hope that at some point in the near future we're able to find a balance that we're relatively and collectively happy with, for the sake of our future.

[throwworhtthrow]

LLMs don't have rights. LLMs are tools, and the state can regulate tools. Humans acting on behalf of these companies can still, if they felt the bizarre desire to, publish assembly instructions for bioweapons on the company blog.

[xscott]

You're confused about whose rights are at stake. It's you, not the LLM, that is being restricted. Your argument is like saying, "Books don't have rights, so the state can censor books."

[nomel]

> if they felt the bizarre desire to, publish assembly instructions for bioweapons on the company blog.

Can they publish them by intentionally putting them into the latent space of an LLM?

What if they make an LLM that can only produce that text? What if they continue training so it contains a second text they intended to publish? And continue to add more? Does the fact that there's a collection change things?

These are genuine questions, and I have no clue what the answers are. It seems strange to treat a implementation of text storage so differently that you lose all rights to that text.

[SilverElfin]

I have rights. I want to use whatever tool or source I want - LLMs, news, Wikipedia, search engines. There’s no acceptable excuse for censorship of any of these, as it violates my rights as an individual.

[logicchains]

>LLMs are tools, and the state can regulate tools

More and more people get information from LLMs. You should be horrified at the idea of giving the state control over what information people can access through them, because going by historical precedent there's 100% chance that the state would use that censorship power against the interests of its citizens.

[cwillu]

“More and more people get information from LLMs” this is the part I'm horrified by.

[miltonlost]

I'd rather be horrified that people are getting information from LLMs when LLMs have no way to know what it's outputting is true.

[cruffle_duffle]

And the government is going to somehow decide what the truth is? Government is the last entity on earth I’d trust to arbitrate the truth.

[next_xibalba]

Are you also horrified how many people get their facts from Wikipedia, given its systematic biases? All tools have their strengths and weaknesses. But letting politicians decide which information is rightthink seems scary.

[nubg]

Was this comment written with the assistance of AI? I am asking seriously, not trying to be snarky.

[davidmckayv]

No. I just write well.

[freedomben]

You clearly already know this, but you do in fact write very well!

[davidmckayv]

Thank you!

[Animats]

> Today it's "catastrophic risks." Tomorrow it's misinformation, hate speech, or whatever else the state decides needs "safety mitigations."

That's the problem.

I'm less worried about catastrophic risks than routine ones. If you want to find out how to do something illegal or dangerous, all an LLM can give you is a digest what's already available on line. Probably with errors.

The US has lots of hate speech, and it's mostly background noise, not a new problem.

"Misinformation" is more of a problem, because the big public LLMs digest the Internet and add authority with their picks. It's adding the authority of Google or Microsoft to bogus info that's a problem. This is a basic task of real journalism - when do you say "X happened", and when do you say "Y says X happened"? LLMs should probably be instructed to err in the direction of "Y says X happened".

"Safety" usually means "less sex". Which, in the age of Pornhub, seems a non-issue, although worrying about it occupies the time of too many people.

An issue that's not being addressed at all here is using AI systems to manipulate customers and provide evasive customer service. That's commercial speech and consumer rights, not First Amendment issues. That should be addressed as a consumer rights thing.

Then there's the issue of an AI as your boss. Like Uber.

[cosmic_cheese]

Presumably things like making sure LLMs don’t do things like encourage self-harm or fuel delusions also falls under “safety”, but probably also “ethics”.

[SilverElfin]

Yep this is absolutely censorship with extra steps but also just an unnecessary bureaucracy. I think the things you have in quote are the core of it - all these artificial labels and categorizations of what is ultimately plain old speech, are trying to provide pathways to violate constitutional rights. California is not new to this game however - look at the absurd lengths they’ve gone to in violating second amendment rights. This is the same playbook.

What is surprising, however, is the timing. Newsom vetoed the previous verison of this bill. Him signing it after Charlie Kirk’s assassination, when there is so much conversation around the importance of free speech, is odd. It reminds me of this recent article:

Everyone’s a Free-Speech Hypocrite by Greg Lukianoff, the president and chief executive of the Foundation for Individual Rights and Expression (FIRE) https://www.nytimes.com/2025/09/23/opinion/consequence-cultu...

[lupusreal]

Good post. It's not even about the rights of the LLM or the corporation, but of the people who will be using these tools.

Imagine if the government went to megaphone manufacturers and demanded that the megaphones never amplify words the government doesn't like. "Megaphones don't have rights so this isn't a problem", the smooth brained internet commenters smugly explain, while the citizens who want to use megaphones find their speech through the tool limited by the governments arbitrary and ever changing decrees.

As for the government having a right to regulate tools, would a regulation that modern printing presses recognize and refuse to print offensive content really fly with you defending this? The foremost contemporary tool for amplifying speech, the press is named right in the first ammendment. "Regulating tools" in a way that happens to restrict the way citizens can use that tool for their own speech is bullshit. This is flagrantly unconstitutional.

[josefritzishere]

I've never thought censorship was a core concern of AI. It's just regurgitating from an LLM. I vehemently oppose censorship but who cares about AI? I just dont see the use-case.

[logicchains]

Censorship of AI has a huge use-case: people get information from AI, and censorship allows the censors to control which information people can access through the AI.

[cruffle_duffle]

Worse people including me easily delegate parts of our thinking to this new LLM thing.