[alach11]

The HN discourse on this subject is incredibly discouraging to me. It's completely nonconstructive.

I take the pace of AI development seriously. All of my friends who work at AI Labs take it seriously. It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.

If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing? We should want the AI Labs to make binding agreements to proceed safely and slower, rather than being stuck in a suboptimal game theoretical race to advance unsafely. We should be hoping that nations can make binding agreements. Our discourse here should be about the substance of what these agreements could be and the practicality of enforcement mechanisms.

[NitpickLawyer]

> We should want the AI Labs to make binding agreements to proceed safely and slower

That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper?

The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.

Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?

[txrx0000]

We have to stop comparing nuclear weapons with general intelligence. They are nothing alike.

Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.

There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.

[0xDEAFBEAD]

>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use.

Perhaps a better analogy, instead of "nukes", would be "nuclear technology".

Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.

Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.

>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.

Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run

We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.

Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.

Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.

Almost all the species which have lived on Earth have gone extinct. It could happen to us too.

[txrx0000]

> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.

The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.

What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.

You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.

And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.

[0xDEAFBEAD]

>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.

It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.

If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?

[0xDEAFBEAD]

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else.

This sort of cynicism feels un-falsifiable.

If companies advocate against regulation, the cynics will say: "These corporations want capitalism to run amok with no regulation to interfere with their profits. They need to go full steam ahead in order to show profits for their investors."

If companies advocate in favor of regulation, the cynics will say: "Regulatory capture."

If you have legitimate concerns, how about a constructive discussion regarding how those concerns could be addressed, in light of the "risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems" which over 1000 frontier AI company employees expressed worry about?

I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.

>The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.

What are you referring to? Are you referring to this quote from the OP?

"The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."

Your response seems to be a total non sequitur, or failure to understand Leo's point. Exponential growth can easily get out of control, as we saw early in COVID. The point about nuclear is mainly an analogy.

[NitpickLawyer]

> which over 1000 frontier AI company employees expressed worry about?

Are they the same as these [1] 700 employees?

> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.

They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.

Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.

[1] - https://www.forbes.com/sites/tylerroush/2023/11/20/more-than...

[0xDEAFBEAD]

This seems like an oversimplification. OpenAI was on top at the time sama was fired as well.

Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.

The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.

[watwut]

Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way. We are cynical, because it is blatantly clear these are not good faith actors. And yes, regulations they are for have an effect of restricting their competition.

> I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.

I would like to see a token effort of good faith engagement from AI companies. For a change.

[0xDEAFBEAD]

>Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way.

I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?

>And yes, regulations they are for have an effect of restricting their competition.

Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?

>I would like to see a token effort of good faith engagement from AI companies. For a change.

What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?

[yellowapple]

> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.

It's very easy to make those dismissals because of how transparently obvious it is that self-interest is a significant factor, no matter how loudly they insist otherwise.

> If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?

Similar attitudes hindered nuclear energy research and deployment for decades, in turn kneecapping our species' ability to obsolesce fossil fuels and avoid the global climate catastrophe we've now made inevitable.

[atleastoptimal]

So many otherwise smart engineers, overrepresented on HN, have this weird mental block where they fail to take AI progress seriously. I think it's a fear that their status-defining skill could be commodified, so they adopt a mental framework that aims to dismantle the seeming legitimacy of any genuine leap in AI progress as some sort of mirage

[Davidzheng]

Not just engineers - I see people attributing it to strong priors that nothing happens but in my opinion it's more some form of psychological defense mechanism against understanding there will be an intelligence vastly superior to humans which is a deeply uncomfortable thought (to many).

[dullcrisp]

“Nothing ever happens” is like “you can’t beat the market.” It’s basically tautologically the strongest predictive model that we can have. A corollary is that something may happen, but it won’t be what you think.

[0xDEAFBEAD]

Interesting fact. Bryan Caplan is a economist who became famous for making bets on the future. He would take the "nothing ever happens" side of the bet, and he would almost always win. In 2023 he took the "nothing ever happens" side of an AI bet. Just 6 months later, he wrote:

"When the answers change, I change my mind [...] Base rates have clearly failed me. I’m not conceding the bet, because I still think there’s a 10-15% chance I win via luck... make no mistake, this software truly is the exception that proves the rule."

https://www.betonit.ai/p/gpt-retakes-my-midterm-and-gets-an

At this point, straightforward trend extrapolation suggests that AI continues to get better and better. Arguably, continued rapid AI progress is the "nothing ever happens" bet at this point. If AI progress stops, or slows dramatically, something will have happened.

[peyton]

Why should I trust these randos instead of Claude? I know Claude. I would hope Claude is smarter than whoever wants to make rules about my use of Claude. Replace Claude with Kimi, etc. if needed.

I don’t find a petition from people I don’t know who don’t care about me more compelling than my own direct experience with these models.

[0xDEAFBEAD]

I suggest you give Claude a link to the petition and ask Claude what it thinks.

(I did so previously and pasted the response here, but it got auto-flagged for AI content.)

[txrx0000]

It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem. Most people here take AI progress very seriously judging by the number of posts on the front page about AI at any given time. And I think some engineers even explicitly want to commodify their "status-defining skill":

https://nitter.net/__tinygrad__

[0xDEAFBEAD]

>It's ironic that the top reply to a comment about quality of discourse on this site is an ad hominem.

You might find the Ad Hominem Fallacy Fallacy page interesting:

"Argumentum ad hominem is the logical fallacy of attempting to undermine a speaker's argument by attacking the speaker instead of addressing the argument. The mere presence of a personal attack does not indicate ad hominem: the attack must be used for the purpose of undermining the argument, or otherwise the logical fallacy isn't there. It is not a logical fallacy to attack someone; the fallacy comes from assuming that a personal attack is also necessarily an attack on that person's arguments."

https://laurencetennant.com/bonds/adhominem.html

So for example, if I said: "The arguments for constructing an AI pause button are bad because they are being made by AI company employees", this would be a fallacious argument.

If I said: "HN users have this weird mental block where they fail to take AI progress seriously", that's not necessarily a fallacious argument, unless it's for the purpose of undermining an argument the HN user made.

[BugsJustFindMe]

> Most people here take AI progress very seriously

You have only an extremely weak basis for this claim and have essentially just made it up.

> judging by the number of posts on the front page

Posts reaching the front page is not a reliable way to measure what "most people here" believe.

> It's ironic that

It's not irony.

[soundworlds]

It really comes down to this: Sam and Dario are leading companies that who made the reckless decision to train on everyone's IP without proper process or consent. It has been proven they trained on pirated content. There is no question about this.

Then, as soon as there are open models who threaten their businesses, they cry "distillation and IP theft".

It's the last 3 years of pure hypocrisy and fear-baiting for investment.

To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.

[0xDEAFBEAD]

You're missing the point of the OP:

"there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems... each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration"

>To be honest, I think people would now prefer the risk of misused open models, when the alternative is giving these guys more power.

Then advocate a muscular regulatory regime which reduces the amount of power they have. Tell them: "You want regulation? OK. How about everyone whose IP you trained on gets some sort of veto option on your models? That should reduce the racing risk that you're talking about."

[soundworlds]

I suppose I have personally lost faith in centralized institutions that would be implementing said regulation, partly because they let OpenAI and Anthropic (along with Suno, Audio, etc etc) get this far uninhibited. Centralized regulation simply can't keep up.

My hopes now lay with a decentralized approach of open models, trusting people to adapt them to local needs and security, rather than a nationwide paternalism.

[0xDEAFBEAD]

I don't exactly see how "trusting people to adapt [open models] to local needs and security" solves the problem of "automating AI research" where "capability development rapidly accelerates beyond our ability to understand or control the resulting systems".

If regulation has been too slow, that's not really an argument for making it even slower?

[philipkglass]

I know a lot more about chemistry and physics than I know about biology, so biological risk is still at least a "maybe" reason to be worried about model safety for me. I don't worry about other weapons of mass destruction developed with LLM assistance. Nuclear weapons are already controlled much more with political pressure and the physical difficulty of e.g. enriching uranium than they are controlled by secret-keeping.

For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

The other justifications for restricting model training don't seem nearly as compelling. Worse, restrictions on model training have an infectious quality if they mean something more than "leading AI labs will voluntarily slow down together." (Like if the signatories mean that better models and software for building them should be regulated.) It's too dangerous to publish stronger models. So it's also too dangerous to publish training software. So it's also too dangerous to publish tools that allow you to build training software. (Claude's Fable won't help you write generic model training pipelines for this very reason.)

If you keep following the implications of this safety argument, it's as broad an assault on the distribution of software and computing as has ever been proposed.

[fidotron]

> For all I know (I don't know a lot about biology), it's possible to develop a virus that spreads as easily as measles, but has the lethality and long induction period of HIV. If that's true, and if LLMs would make it easy to develop such a virus, I see how no-guardrails LLMs could be truly disastrous.

This is the specific example in the latest doomerist wave I have so much trouble with: by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

Similarly, how are you supposed to magically get access to everything required to actually manufacture the offensive organism in question? Are we all doing CRISPR in our kitchens now?

If you are that concerned about this threat then . . . monitor biolabs, not AI. It is so detached from the real state of things as to be bizarre.

[alach11]

I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

Developing a bioweapon is already within reach for experts with sufficient funds, time, and motivation. That's more or less what gain-of-function research already does! Fortunately the barrier to entry is high enough that you don't see like... Boko Haram doing this. Or a lone-wolf nutjob. LLMs could definitely lower that barrier to entry though.

[fidotron]

> I think some AI safety advocates have been arguing for great controls around nucleotide synthesis (e.g., impose greater data collection requirements or advanced screening for harmful sequences). It's surprisingly easy to order synthesized genes delivered to your doorstop. All you need is a credit card and a nucleotide sequence file!

So why have the discussion about regulating AI at all if that is the actual problem? Surely instead of blanket blocking everyone's AI models it makes more sense to control the potentially problematic sequencing?

[pu_pe]

Because there are many legitimate uses for nucleotide synthesis, and we would have to slow scientists down by adding more regulation. The other problem is that it's not like it's very difficult to do it, so a motivated person could still do it even without access to these services.

[intorio]

> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?

[fidotron]

Have you tried to make LLMs do that? Unless it's implementing previously developed algorithms they screw it up so repeatedly it's incredible. (And even when it is clearly defined it's obviously their weak spot).

The "rapid self improvement" is going to be ever further down the LLM rabbit hole where brute force, tenacity, and symbol juggling beat having a reasonable model of anything else. This is enormously more likely to recommend improvements to its own architecture than to develop a sudden grasp of molecular biology.

There isn't an AI bubble, but I'm increasingly convinced there is an LLM bubble, and that this noise is to do with preventing people noticing that before they've managed to pull something else out of the oven.

[BugsJustFindMe]

> We know they cannot do spatial reasoning

Everything else aside, and regardless of whether that claim is currently true, I would ask you to be aware that that statement requires a way of thinking that ignores predictable future developments. Whatever the capability of any systems currently released, there will be more capable ones tomorrow, and the most you can say is that we have not mastered training spatial reasoning yet, not that we can't or won't.

> Are we all doing CRISPR in our kitchens now?

https://www.the-odin.com/crispr-kit/

[Smaug123]

> cannot do spatial reasoning

Opus 4.8 was the first model that could plausibly roleplay a BJJ roll without making a physically implausible move within five conversational turns. Fable 5 was the first model that beat me at the task, where I was the first party to write down an implausible move.

[OkayPhysicist]

You can do CRISPR in your kitchen. It's been possible practically as long as CRISPR has been a thing. It's not particularly difficult, to the point that my biggest counterpoint to the whole AI doomerism argument on the subject is that nobodies done it yet.

You don't need some genius AI to design a bioweapon, you need to do a bit of light reading on bioengineering. The fact that some mad scientist hasn't ended civilization is evidence of the fact that it's not some existential threat that justifies wild levels of government control.

Unfortunately, AI development is dominated by an even mix of 1) Machine God Cultists, 2) Morally Bankrupt Mercenaries. All three agree on the fact that "thou shalt have no gods before me", because it'll make them richer than sin, and allow them to shape the AI landscape in their image.

[sm-silversight]

Can you point me at this light reading? I don't wanna do CRISPR in my kitchen, but itd be cool to know more about it. I thought I needed to be basically pre-med to grasp it.

[SturgeonsLaw]

Here's a step by step guide on how to gene edit rice

https://bio-protocol.org/en/bpdetail?id=1225&type=0

Mostly it's just heating and cooling tiny volumes of liquids

[areoform]

This experiment has been run, and even the biological stuff is far fetched.

I've written about this before, but the issue is that these people have worked with computers their entire lives, and they keep projecting outwards from there.

The 20th century was dominated by mad scientists (mostly Teller) and generals (bombs away LeMay), but one thing that I respect about them is that they didn't just sit and guess about probable futures. They sat down and actually ran experiments.

For example, in the 1960s, there was a whole generation of people asking "who's next?" after the US, USSR, the UK and France had made The Bomb. And instead of just gesticulating wildly and trying to keep fighting the losing fight of "classify everything, admit nothing," Teller found three smart, young physicists (postdocs) with 0 nuclear weapons experience, and indeed 0 weapons experience, and ran an experiment called the Nth Country Experiment, where they were were asked to make a design for a working bomb.

After 2+ years, they were able to. And so proliferation work shifted from controlling knowledge about the technology and the technology itself to materials.

Something similar was done with bioweapons via Project Bacchus, where they gave actual bioweapons experts carte blanche to set up a secret bioweapons lab with COTS equipment. They succeeded. This was in the early 2000s.

This then led to surveillance of specific purchase combinations and equipment. Because tbh, most weapons of mass destruction are commodity technology. Nuclear weapons are 80+ years old. Chemical weapons are over a century old. Bioweapons are god knows how much older. And it's not the knowledge of these things that matter, but intent, materials, and the ability to create them.

Just because you know something about a thing doesn't mean that you're capable of doing that thing.

Let's take bioweapons.

They keep comparing wet work in a lab to writing code on a computer.

When you screw up an exploit, you fail to execute the exploit. Famously, just like software's near zero marginal cost of distribution, the marginal cost of failure is nearly zero.

You can screw up an infinite number of times on your way to a successful exploit.

If you screw up with lethal agents in a lab? You die.

Here's a non-exhaustive list,

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

   A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

   Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

   Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. The accident occurred at the Scientific-Production Association "Vektor" (today the State Research Center of Virology and Biotechnology "Vektor") in Koltsovo, USSR (today Russia).

"lethally infected with the Marburg virus after accidentally pricking himself"

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

And this, of course, doesn't even cover the cost of equipment, the precursors, sourcing the highly specific materials needed, then setting the equipment up... etc.

The same is true for the Bosch-Haber / Haber-Bosch process, which famously made WW1 possible. Every HS'er learns about the process and the steps. Steps that were classified once upon a time and were the subject of negotiation at the Versailles.

Does that mean a HS'er (or any adult) can set up an experiment that works at 177 times the pressure of the Earth's atmosphere to do anything at any scale without significant infrastructure and help?

No, it doesn't work like that.

The people who can do this are domain experts, and they've been able to do this with COTS stuff since the 1990s, at the very least, for a price of around $2M. And those people don't need a LLM to tell them what to do. In fact, they're the exact people who'll have access to unrestricted versions of these LLMs.

And from a security perspective, I would bet good money that flooding the FBI's tip line with junk about every teenager trying to learn "what be a mitochondria" does more harm to the effort of finding people who could be planning such a thing than it helps. It takes more resources to go through the mass of false negatives that have now been created as matter of policy.

These experiments have been run. And we can run them again.

The fictional scenario of someone learning how bioweapons work and conjuring up a plague isn't real and it hurts humanity as a whole to impede the sciences over it.

Because what someone can flail around in / do is learn about immunology / try to "cure cancer" with a LLM and hopefully get started on a long career in medicine. Or, a discovery that matters.

Because in those cases, if and when they do end up at a lab, screwing up doesn't mean death. Just tons of wasted time (and money). And they will fail / screw up. Just look at literally every undergrad in any lab and the expensive messes they create.

[versteegen]

Thank you for taking this seriously enough to write this, and anyone else likewise.

But this is attacking a strawman, amateur bioterrorists. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.

Being at risk of killing yourself also just makes success X% less likely, but if X < 90 that doesn't mitigate much.

[red_green_yell]

It sounds like your claim is "AI is a force multiplier, therefore it is dangerous and must be regulated." Not trying to strawman - I read your comment twice and I think that's what you're saying.

By that logic Excel, the internet, calculators, and air conditioning are all force multipliers in that all of the make it easier for an expert who can make bioweapons able to do so more quickly / comfortably / easily. Obviously it wouldn't make sense to "pace" any of these technologies. Even social media or email could be considered force multiplier in that they could literally allow you to increase the number of collaborators working on supposed bioweapon. Credit cards and modern logistics force multiply your ability to purchase precursors.

Also it would be a force multiplier on the prevention/mitigation side as well. Police armed with AI to scan for suspicious precursor purchases or review security camera footage after the fact would be far more effective than before. It's not obvious that the force multiplication is greater to the bad actor or to those who oppose them.

[versteegen]

I didn't argue "must be regulated". I'm arguing AI is powerful (at achieving things, and hence has dual-use dangers). Many people won't even admit that, which is the part that really annoys me: they don't even want to have a conversation about risks because somehow AI is not actually a powerful general-purpose tool. Of course everything you said is true, though many of those things aren't very powerful. But the internet and social media and smartphones certainly have been of great utility for terrorism and child exploitation (and probably also causing many would-be-terrorists to get picked up).

[areoform]

    But this seems like it's attacking a strawman. AI is a force multiplier in the hands of an expert. If it took a team before, maybe a single malicious actor can accomplish it now that AI can fill in the parts they aren't well-versed in. And that dramatically increases the chance of it happening.

I'm glad that you brought that up, genuinely so.

I want to ask you a question, do you feel like that this is the problem being solved by the current "safety features?"

Let me rephrase my question, do Anthropic and OpenAI implement the same "guardrails" and "safety features" for the NSA? Does the Mythos NSA Edition™ have these restrictions? What about the one that's running as a part of Maven?

How does stopping me from asking about rabbit sex protect you from nut jobs with a security clearance using these systems to go off and make weapons?

Because this has happened before. The only successful bioweapons attack in US history was done by a guy who worked at Fort Detrick. https://en.wikipedia.org/wiki/2001_anthrax_attacks

The only private organization successful enough to make biological and chemical weapons is Aum Shinrikyo and they had university affiliation and nearly a billion dollars in the 1990s.

The guy who led Aum's bioweapons program was

    Seiichi Endo was born in 1961 and attended Obihiro university of Agricultural and Veterinary Medicine. After graduating, he became a student at the Kyoto university medical school research department focusing on AIDS-related gene research at the viral research center. He joined Aum in 1987, became a monk in 1988 and later led Aum’s biological weapons program.

and,

    Among some of Japan's "best and brightest" who joined the cult included a former researcher of the National Space Development Agency of Japan, an expert on chemical weapons who majored in organic physics at Tsukuba University, a researcher who studied elementary particles, a reporter with a major Japanese newspaper, a physicist from Osaka University, a cardiac specialist, and an organic chemist, to name a few.

https://irp.fas.org/congress/1995_rpt/aum/part04.htm

and,

    Through its network, Aum Shinrikyo successfully recruited over 300 scientists and engineers—including employees at the Kurchatov Institute, the premier nuclear facility in Russia—who were either attracted to the apocalyptic ideology or lured with financial incentives.50 Aum Shinrikyo’s Russian contacts enabled group members to access black market materials and hardware.

and,

    In response, the cult constructed the $30 million USD Satyan 7 facility which was equipped with three laboratories, a computer control center, and five reactors, all of which was made from corrosion-resistant Hastelloy, ideal for the production of chemical weapons

Why didn't anyone ask questions about a known cult importing an absurd amount of precursors, industrial-grade HEPA filters, fermenters, and lab equipment?

It's because they'd recruited members of the military, bribed local policemen and politicians, and used their sway to silence critics (or kill them).

How does stopping me from asking Claude about rabbit sex stop people like them?

From where I'm standing, Anthropic and OpenAI would have sold Aum a subscription.

The Fort Detrick guy would have access via the US Government and Aum had university affiliation. I am yet to read a serious proposal that actually deals with these risks and the other real risks of this technology.

[versteegen]

Wow, remarkable. Clearly Aum is a different league from the lone-wolf "Fort Detrick guy", treat the risks separately. But I'll take these questions as rhetorical. (See my reply to the sibling comment.) I can't answer them and I'm not defending the guardrails on Claude; in their current form I too find them pretty ridiculous.

[azeirah]

Please write up your comments as blog posts, articles or whatever you can do.

There is such an incredible amount of bullshit going on in the conversation. You are contributing a point that is necessary for people to start taking into consideration and the only way to get people to talk about it is by repeating it 100 times.

I fear that the AI labs will get their way and we'll get idiotic restrictions that favor them, all in the name of "security" when the reality is they care about power for themselves, not real world security.

[testerteert000a]

> ...I see how no-guardrails LLMs could be truly disastrous...

by all accounts this is what's what because of how...

> You can do CRISPR in your kitchen.

.. and because of the same dynamic we see occurring in cybersec.

If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.

If you're good enough at biology and have certain ideologies, but not good-good at specific skillsets like genetic engineering in your kitchen, do LLMs get you good enough? Does the impact on cybersec transfer over to this domain? H

Hard to say, but it has enough risk indicators for me to think the pushback against these possibilities is the height of otherwise talented engineers being myopic.

If good enough at biology can include all the PhD programs across all the world, and you pair other known events like cartels and terrorism groups hiring chemistry PhDs from University of Unemployed Chemistry PhDs, then the next logical extension of that theme with this tech is intense.

[red_green_yell]

If the accuracy of your biology claim is "hard to say" then why would pushback be self-evidently myopic?

You and others are making a claim about LLMs enabling biological dangers. Some people disagree with you, pointing out that the incremental knowledge an LLM provides over say google or a library card may not be that much.

What seems myopic to me is un-skeptically accepting "AI dangerous, must regulate" messaging from frontier labs that have trillions of dollars of vested interests in this message.

[0xDEAFBEAD]

>frontier labs that have trillions of dollars of vested interests in this message.

OP suggests the opposite: "each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration."

[areoform]

    If you're good enough at cybersec, but not good-good at specific skillsets like offensive, then LLMs and a comp sci degree makes you quite capable in the ways that count. This is a fact, full stop.

Wet work in a lab isn't the same as writing code on a computer.

If you screw up with a lethal pathogen in a lab, you die.

If you screw up with code on a computer, you go get lunch.

They're not the same.

[overfeed]

> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest.

It's not a shallow dismissal if you have been following their - let's charitably call it "evolving" - public posture on AI legislation.

It wasn't long ago the same people/companies successfully lobbied the Trump admin to quash state laws that threatened to "Pace" AI, and set ground-rules for safety. Back then, open source models were very far behind the frontier, and their spiel then was it was imperative not to impinge on AI progress with red-tape, lest America's adversaries catch up.

I challenge you to explain what triggered the about-face. The simplest explanation is that open source/Chinese models have caught up with the frontier, and the change in opinion is tactical (and obviously self-interested), rather than a deeply held belief.

[red_green_yell]

Are they lazy dismissals? Or a simple observation that LLM scaling is hitting diminishing returns way way short of AGI.

If the claim is (1) AGI coming and (2) we should so something about AGI, wouldn't it be smart to confirm (1) before worrying about (2)? Especially if certain companies have trillions of dollars of valuation that depend on society as a whole believing (1)?

[roywiggins]

What if "way short of AGI" is still dangerous in ways we can't easily handle?

[overfeed]

When did the pacers have this mass-revelation about dangers of AI? The timing is curious.

Which takes priority: public safety or the free-market + national security? The AI labs hype in now biting them in the ass: why would China and the rest of the world deny their businesses a technology that "is bigger than the invention of electricity" or whatever hyperbole the snake-oil salesmen came up with when they were not breathlessly declaring that the first side to get AGI wins forever.

Even if the pacers are right, it's like greenhouse emissions all over again; each country wants everyone else to cut emissions, but no one wants the economic hit of doing so.

[bpascal]

i think part of the problem, though, is that we've already hit many of the warning signs that AGI is coming, and people have kept moving the goalposts. if you were to tell someone from ten years that there's an ai that has passed the turing test and has made novel mathematical discoveries, i think many would say AGI is already here.

[versteegen]

The old observation that people tend to define AI as whatever computers can't do yet is as true as ever. It's getting a bit absurd, moving from demanding "general intelligence" to replicating human cognitive phenomenology (the experience of cognitive activities). Yet LLMs can already somewhat (confabulation-prone) introspect their own "internal unspoken thoughts" in their residual streams, quite fascinating.

[marcus_holmes]

You're right, in that we would have said that.

I wouldn't frame it as moving the goalposts in a bad way, though. We understand more now than we did then, so we have a better idea of what the actual goal is, and moving the goalposts to frame that goal is more rational than insisting the goalposts stay where they were ten years ago even though the entire pitch has changed since then. I'm straining the analogy, obviously.

[gr_norm]

Moving the goalposts, so to speak, is how science works! We must of course update the things we think based on an improved understanding of how the world works. Only the deeply incurious could consistently demand that one adhere dogmatically to a prior way of understanding the world in the face of changing knowledge. It's important not to treat science as a game to be won (of which 'goalposts' are evocative), but as the collaborative effort that it is.

Note well that I have not said LLMs are not useful, only that no amount of interaction with them has convinced me they remotely approach general intelligence. Their actual function of heuristically regurgitating all information ever known to mankind remains extremely useful in lots of domains.

And by the way, it's not obvious an LLM could fool me (or the average person) over a sufficiently long period of time, exactly because they are heuristic machines. That they lack thought guided by underlying cognitive models seems to always leak out, in the end. I suspect pass rates by LLMs on long-range Turing tests (if there have been any conducted) might drop as people become acclimatized to them.

[bpascal]

what do you mean by 'general intelligence'? i definitely agree that in comparison to humans, they have some definite weaknesses. however, i think they also have definite strengths. and i don't think their current weaknesses imply that much about a potential singularity or ai risk.

[gr_norm]

Sorry, I should've used clearer language. I just mean intelligence in the sense humans possess it. Computers have always been able to exceed limited elements of human intelligence (say, at arithmetic), but the problem is to simultaneously match or exceed all of them. I specifically think that the important parts they currently lack make them a no-go for supposed existential risk.

[JackFr]

If that’s truly the case, why aren’t they offering to be absorbed by national labs. That’s what we do for dangerous biological and nuclear work.

Oh wait they want options and shares and to get rich.

And there’s no natural moat. Which Federal regulation would bring.

That’s why people think it’s disingenuous.

[dmarcos]

Every single technology in history has had people claiming potential existential risks that didn’t materialize. Burden of proof is on people making the risk claims. Those have to be specific and unique to that technology to grant special actions.

[bpascal]

i'm not trying to be rude, but i really don't get what you mean by this? (https://www.amazon.com/Superintelligence-Dangers-Strategies-...) there's been people making specific and unique risks to this technologies for years, before AI was even present. anthropic was literally founded because they were worried about existential risk? also, previous technologies have had existential risks? do you deny that nuclear weapons could have killed humanity? obviously, there have no been existential risks that have killed all of humanity, because by definition, that can only happen once. it will literally be impossible for someone to ever point to an example of that happening, because they will be dead by the time that happens.

[red_green_yell]

See printing press and internet!

[dmarcos]

And video games, democracy, novels, TV, cars, photography, cryptography, the Beatles… pick your favorite and read commentary / opinions of the time. Many thought that came with unique risks never seen before and must be stopped!

[aesthesia]

How many of those concerns came from the people building the technologies? John Carmack wasn't up in arms about how Doom was going to destroy society. Charles Dickens didn't think novels rotted the brain. Johannes Gutenberg wasn't concerned about the effects of a free press.

[dmarcos]

None of the previous concerns were true and nothing indicates current ones are. It’s extrapolation, hypothetical apocalyptic imagined scenarios and piles of assumptions on how the technology could evolve. What makes even less credible in the AI case is that there’s financial / power to be gained by those making the claims.

And to your point. Don Quixote, the first modern and greatest novel of all times, is about a guy that lost touch with reality from reading too many books

[ascorbic]

None of the previous concerns were true, largely because they came from laypeople who didn't understand the technology. A prominent exception is nuclear weapons, where their creators did worry their creations could end up destroying the world. I think most people would agree those concerns were justified, and it's not any lack of capability that means we're still here.

The motivations aren't unique here. Restrictions on the development of almost any technology benefits incumbents by providing a moat.

[aesthesia]

I mean, Cervantes was parodying medieval romances. That's what Don Quixote was reading, not modern novels.

The point is, it should give you at least some pause when the people most familiar with a technology are among those making the strongest statements about its potential dangers. Extrapolation has worked extremely well in predicting what AI will be capable of over the past 5 years; how confident are you that it will fail before we reach apocalyptic scenarios?

[tpm]

> All of my friends who work at AI Labs take it seriously. > If AI does pose biological risk, or could destabilize geopolitics, or even just poses economic risks, surely a joint petition like this is a good thing?

If they seriously think what you just wrote then they should follow through with quitting en masse because that's currently the only way to slow the progress towards oblivion. Are they quitting?

> We should be hoping that nations can make binding agreements.

The big AI Labs are making sure international binding agreements are followed by financially supporting the notorious binding agreement creator and follower Donald Trump. That's how seriously they take this issue.

Yeah let's have a constructive discourse and let's start that with dropping all hypocrisy, shall we.

[ubertaco]

We're commenting on a site run by a Silicon Valley tech start-up accelerator. "Move fast and break things" is the accelerationist way, even if the "things" they break are a stable, functioning social order. As long as you IPO and cash out along the way, you can wreck everyone else's lives just fine, in the minds of this sort.

[alexashka]

> It's very easy to make lazy, shallow dismissals that leaders like Dario/Demis/Sam are acting purely out of self-interest

Pointing out the endless stream of lies and self serving half truths these terrible people engage in helps the younger generation understand the world they were born into.

It's constructive and helpful. It's why I'm replying to you.

You interpreting it as 'shallow dismissal' is shallow dismissal on your end. You're the shallow dismisser, not the people calling terrible people terrible.

As for your little petition. It's as good as eliminating plastic straws to save the environment.

Terrible people with power they don't deserve doing pathetic little, tiny 'good' deeds to sleep better at night needs no explaining.

[anon373839]

Agree with everything you wrote here. Every word. The GP strikes me as unbelievably credulous and trusting of the “leaders” in this space.

I cannot fathom having that kind of unquestioning faith in people who have done absolutely nothing to deserve it.

[potsandpans]

What happens when a country doesn't agree to play along with this little decel game?

The genie is already out of the bottle and now a bunch of people who helped release it think that the collective powers of the world are just going to put it back in.

How far are the saftyists willing to go? Bombing data centers? Starting wars? Embargos?

It's a terrible dilemma to be in. If the threats are real, it gives all the more incentive for some actor to race and try to grab it first. If not, then any punishment isn't proportional. Explain to a nations citizens under a 10 year embargo, "sorry about your quality of life, we all just got a little spooked by science fiction for a second."

Folks try to reach for bad analogies: nuclear bombs and biological weapons. But conveniently omit that by the time the regulations took place for these things we had unequivocal proof of their destruction, scope and scale.

So, what then? We're supposed to collectively stop innovating and put the keys to the kingdom in the hands of a few capitalists, a corrupt is administration and the ccp?

I reject that these agreements should take place, on the grounds that the proposal is delusional and not founded in reality.

[llm_nerd]

On the geopolitics, the American domiciled firms are completely in bed with the US military and US government, and have made it absolutely clear that they are committed to the domination of the US government. So when you say "geopolitical destabilization", you mean "only the US government is allowed to destabilize, which they have done for decades and wish to continue doing unfettered."

And given what we have seen from that government -- piracy, terrorism, war-mongering, thievery, and just profound stupidity -- no thanks, in that case I'd like to see every American AI company utterly obliterated by any other country. If China becomes utterly dominate in AI, awesome.

[rossant]

This is serious. We're in uncharted territory. The geopolitical context is already complicated enough that we shouldn't be dismissive of such a rapid technological race. I'm glad AI leaders are taking it seriously. I'm surprised so many HN commenters aren't.

On the other hand, I'm pessimistic about initiatives like this being effective on a global scale (though they could certainly work at the national level).

[nostrebored]

we're always in uncharted territory. that's why most HN commenters think that this is hyperbole.

[ls612]

"Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place. And some said that even the trees had been a bad move, and that no one should ever have left the oceans."

[txrx0000]

There is constructive discourse, but this kind of commentary doesn't get to the top:

https://news.ycombinator.com/item?id=49078376

https://news.ycombinator.com/item?id=49077753

It's not a given that AI poses bio/political/cyber/economic risk in a way that warrants government intervention. Discourse should first revolve around whether it makes sense for the government to step in at all.

[GardenLetter27]

As a Europoor I've seen the devastating consequences of rampant safetyism first hand. Entire industries and energy security destroyed to satisfy the precautionary principle.

We need to accelerate as much as possible and create abundance as fast as we can. Life is short.

[0xDEAFBEAD]

"This building is too cold. Time to set it on fire."

Let's not swing from one extreme to the other.

If American hypercapitalists, living in Silicon Valley of all places, tell you that you need to regulate, maybe regulation really is called for in these specific circumstances.

[anon373839]

https://en.wikipedia.org/wiki/Regulatory_capture

[0xDEAFBEAD]

Well, we've been talking about the EU. EU regulation has not been especially favorable to US big tech. The EU is collecting more from fining US big tech than it collects in taxes from EU public tech companies:

https://xcancel.com/levelsio/status/2080314960656159018

I think you should at least consider the possibility that this letter is not primarily motivated by regulatory capture, and insiders at these AI companies realize something which outsiders don't.

To me, the regulatory capture theory predicts that the only signatories on the letter should be big AI firms which can expect regulatory influence. But there are a number of signatories from smaller firms such as Thinking Machines, Core Automation, Inherent, and Prime Intellect.