[smokel]

> The path to AGI (Artificial General Intelligence) must never be enclosed by high walls. We have always believed that AGI should be the cornerstone for all of humanity to collaboratively explore the boundaries of intelligence and solve complex challenges, rather than a privilege monopolized by a few rules and subject to revocation at any moment.

This is not obvious to me. If everyone gets access to AGI, but only a few people have the means to do really bad things with it, then what is the difference? Might as well make clear from the start that AGI is a powerful tool (read: weapon), and not a solution (e.g. world peace).

[hedora]

The printing press gave us the renaissance, even though the church argued it was too dangerous to give non-clergy access to books.

Even things like universal access to guns was a net positive. It led to the end of feudalism and rise of democracy.

The sad truth is that whenever any one group of people gets a monopoly over an important technology, they use it to exploit/enslave/murder everyone they can. Look at the international news for examples from 2026.

[reedlaw]

Since the Renaissance got started before the printing press, maybe you mean the press fueled it? The idea that the church found printing dangerous seems like a conflation with events that happened during the Protestant Reformation. The Catholic Church did censor works it found heretical, including unauthorized Bible translations.

One could argue the opposite conclusion, that technology helps break monopolies, but either view depends on reductionist historical readings. The truth is somewhere in between.

[enugu]

Restricting things like creation of a highly infectious virus is very different from restricting books or even guns. There is no 'monopoly' over such a technology, as a use of the technology will inevitably harm the creators themselves.

Restrictions on high end biology, chemistry would leave overwhelming number of use cases of LLMs unaffected - no need to ban open weight LLMs. Such restrictions can be even more effective, if it is coupled to researchers getting early access to see the possible problems and have an opportunity to prevent the outbreak or create new vaccines well in advance.

Restrictions are not enabling monopolies. The opposite is true, if a LLM engineered virus or other harmful technology is let loose, public opinion can very quickly swing towards draconian regulation. (see nuclear power after Chernobyl).

[somenameforme]

Speaking practically your hypothetical is a scenario that requires somebody that is proactively interested in, and theoretically capable of, making a e.g. dangerous virus, yet are unwilling/unable to do so without a chatbot. How many people might this possibly apply to? I think the number is literally zero.

I also don't entirely understand your comment, because your latter parts do not follow from your lead. You're 100% right that somebody who's not extremely capable messing with this stuff is overwhelmingly likely to just hurt themselves. And somebody relying on a chatbot to guide them in dealing with this sort of tech? Yeah, they're gonna win a Darwin Award.

---

I also think there's an entirely different, yet also compelling argument, against censorship. Local LLMs already exist and are advancing rapidly. There will come a time, probably in the relatively near future, when the state of the art big system and a decent uncensored local system will become practically indistinguishable in terms of capability. So not only will people be able to do this locally, but you lose something big in the process.

The reality is that our interactions with LLMs are 100% being actively surveilled, regardless of privacy promises of the companies involved. At the minimum, every chat is making it's way over to the NSA's Utah data center, one way or the other. Some guy trying to do something significantly malicious using an LLM is little more than a gift to the authorities, but this is only true with centralized/online uncensored services. Push people onto local models to do nefarious stuff, and law enforcement is blinding themselves.

[munksbeer]

>Speaking practically your hypothetical is a scenario that requires somebody that is proactively interested in, and theoretically capable of, making a e.g. dangerous virus, yet are unwilling/unable to do so without a chatbot. How many people might this possibly apply to? I think the number is literally zero.

I don't disagree with the rest of your post, but this doesn't seem correct.

I think I'd phrase it that there probably already exist, or will exist, people with the inclination to cause global mass death, but don't have the knowledge or ability to manufacture a virus to achieve this.

[somenameforme]

The important part is being theoretically capable of. Fortunately there are massive barriers to doing things like synthesizing deadly viruses, and it's not just a matter of knowledge but of skill. For instance there was a Japanese death cult [1] that at its peak included not only many graduates of top universities in Japan but tens of millions of dollars in funding. But their escapades read a lot like a satire of incompetence.

That's not to say they were harmless - they managed to kill numerous people, but they'd have killed vastly more if they just drove some trucks into crowds as is becoming a typical weapon of terrorists. And I think the main reason is because knowing how something is done, and actually doing that thing, are radically different.

For a goofy analog, think about assembling sofas or even certain desks/chairs from a kit. That can actually be fairly tricky, to the point that there's an industry built around doing it for you. But there it's literally following like a few dozen steps with a carefully manufactured set of goodies and all tools right in front of you. Imagine doing something many orders of magnitude more complex where you're improving everything, have guidance that may be simply wrong, requires not only extreme skill but also a wide variety of difficult to acquire equipment, and if you make any mistake - you stand a decent chance of killing yourself.

[1] - https://en.wikipedia.org/wiki/Aum_Shinrikyo

[enugu]

If it just a mundane chatbot, the discussion is moot. But, we already have AI making breakthroughs in research and approaching the abilities do science just like a scientist does. (The last two paragraphs of your comment also assume such a high capability scenario).

Imagine giving the access, to whoever wants it, to a scientist who may not have many fresh insights, but has the advantage of a huge memory containing all the scientific literature in their mind, the standard patterns of deductions, and the ability to work at a very fast pace 24/7. They could identify vulnerabilities in biological mechanisms, just like AI identifies security flaws in code today.

---

Regarding hurting themselves, I was not referring to someone who is too dumb to follow lab safety precautions, but someone who has a nihilistic mindset. State actors and militia use weapons to take over and enjoy the power they acquire - they dont want to get killed by a deadly virus(unless they engineer and selectively apply the vaccine before they release the weapon - but this is very hard to keep secret). Someone who is nihilistic wont have such reservations on using the weapon even if it destroys them eventually.

Regarding restrictions on API LLMs leading to use of local LLMs, it is the local LLMs which will be used anyway (once they have the capability). That we live in a mass surveillance envirnoment is common knowledge. The bottleneck, where restrictions can be applied, is not inference but training which requires hundreds of millions of dollars. Chinese scientists have themselves spoken about AI safety concerns and it is indeed a threat to China just like anyone else.

Also, restricting high end weapons ability does not interfere with 99.9% of LLM usage (open-weights or proprietary) - so it need not interfere with business strategy.

[sterlind]

I'm amazed we didn't have the same moral panic when the web became popular. billions of people suddenly had access to knowledge about how to create dangerous viruses! sites like Wikipedia don't even check that you're a US citizen before letting you access pages on recombinant DNA and genetic engineering! the articles on sarin and VX nerve gas include syntheses!

[enugu]

Wikipedia is a presentation of partial selection of biology textbooks and research papers, not using them as a collective brain to generate new artifacts.

There is a big difference between having a large bookshelf of programming language/networking/OS manuals and the ability to generate a functional software product which previously required a hundred or more developers. Even a hundred developers may not be able to find a subtle exploit in code which requires a tedious scan of millions of lines. Computer security hacks can be much less of a problem in comparison to exploits in biology.

Also, even Wikipedia (and public resources in general) have restrictions - there is information dangerous enough to be not published. In the 1930's itself, Szilard (who discovered the chain reaction) and Bohr advocated for restrictions on openly publishing research on uranium fission.

[airstrike]

Restricting access helps even less.

And none of this is AGI so...

[allarm]

How do you define AGI these days?

[airstrike]

I don't have a fully perfect definition, but I can name a couple of requirements.

Ironically, both reasoning and agency are required, neither of which our "reasoning agents" possess.

[mapontosevenths]

Are you unironically claiming that LLM's can't reason? That's an absolutely wild claim in an era where they're solving Erdos problems and writing better code than many senior devs. What's the basis for it?

Agency is harder to define, but most any definition I can come up with LLM's meet. Again, I'm curious how you define it in a way that excludes frontier models but doesn't also exclude many humans.

[airstrike]

Yes, unironically claiming that and not wild at all if you're a practitioner.

It doesn't become actual reasoning just because you chose to call it so. If they did reason, LLMs would not fail at ridiculously easy problems like strawberry or car wash ones.

LLMs are great at search. They only emulate reasoning. They can't actually reason but they approximate it. Combine it with copious amount of computes and some search problems become tractable.

[mapontosevenths]

> They only emulate reasoning.

If they emulate reasoning well enough that it gets the same or better results what is the difference? Semantics? I can't help but wonder if you dont percieve what they do as reasoning because its different from the way you reason?

> strawberry or car wash ones.

Humans fall for the Nigerian scam still. We all have blind spots but that doesnt imply we're all completely blind.

[smokel]

If humans did actual reasoning, then why is this particular discussion failing so hard?

[munksbeer]

> They only emulate reasoning.

So do we.

[allarm]

> I don't have a fully perfect definition

It feels like no one has. Am I wrong? How can we even talk about something that doesn't have a definition?