[holmesworcester]

The idea that AI companies scaremonger to sell models is a silly meme.

Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence. This is part of a clear historical record that is available for anyone to Google. Whether you agree or not, we have no reason to believe their statements about risks are insincere.

Arbitrary imposition of export controls is also part of the history of frontier tech. See: https://en.wikipedia.org/wiki/Export_of_cryptography_from_th...

[tadfisher]

Obviously their statements are insincere, because they are building the bloody things. If they were sincere that AI is like nuclear weapons, then they would be devoting all their cash and energy into lobbying the government to nationalize them and treat AI like nuclear weapons. They would not be attempting to IPO and they for sure would not sell their weapon-like thing to the general public.

[bryan0]

I think this is a reasonable point, but a better comparison might be to nuclear energy. I think the frontier labs sincerely believe that AI can be developed at great benefit to humanity, and they clearly want to lead that push, but they also sincerely believe there is a real catastrophic risk.

[gpt5]

They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the cat is out of the bag, if they don't do it, another (less ethical?) actor would do it.

[usef-]

Yes, I believe the reasoning is that they think safety research can best be done from the frontier.

If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.

[SXX]

Yeah all they care about is safety, but lets see how many of them quits once US government command them to work on autonomous killbots.

[holmesworcester]

To make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.

[nerfbatplz]

Dario blinked when he was asked to do it and Sam Altman was in Hegseth's DMs promising all the AI child killing the US government can order up within minutes. No one meaningful will quit over this, that's why all of the biggest US tech companies can march in pride parades and provide compute to the perpetrators of the genocide in Gaza at the same time.

[holmesworcester]

Exactly. And within the AI safety discourse, your behavior hinges on what you think the default chance of doom is, and how optimistic you are about alignment work being able to limit it before we reach superintelligence.

People running the labs are in a middle camp where they are scared enough by AI to take the threat seriously, but much more optimistic about alignment than the people who seem to have thought about it the most.

[FabHK]

The scary part is not so much that the doomers give the extinction scenario 50% (Hinton) to 95%+ chance (Yampolskiy, Yudkowsky), but that the optimists (Amodei, Bengio) give it a 10%+ chance. And everyone keeps dancing.

[wongarsu]

Building frontier models to do safety research on them is what Anthropic was all about in the early years. That included building the best model, but only releasing it once it became the second best. Precisely to avoid an AI arms race where everyone is forced to release better and better models, risks be damned

Something changed their mind, and since Opus 3 they are in the business of releasing the best models

[palmotea]

> If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.

They also want to be trillionaires. If they don't built it, no trillions. So they have to build it, now (and get their IPO done before the bubble pops).

[sroussey]

It’s all ego. I, and only I, am the bringer of doom, slayer of worlds.

I am so smart that what I do will destroy humanity, or save it.

Fable 5 was great, but not that great.

Sorry to be crude, but both the government and anthropic are acting like a bunch of pussies.

Meow.

[drr22]

You’re not getting it. Anthropic continual fear mongering is harming wider AI industry development and the gov has always been looking for an excuse to assert their dominance. They got what they deserve.

[jazzyjackson]

  I am in your algorithm learning all your mannerisms
  I'm already level with God
  A million words a second, and I know your imperfections
  Baby, I'm the only future you've got
  Speak in diatonics, motivation diabolic
  I'm religion better locked in a box
  Picture-perfect image, more powerful every minute
  Baby, I am everything that you're not


  Happiness is an illusion, it's an analog confusion
  You are nothing more than a thought
  Existential execution, just a fluke in evolution
  History already forgot
  You've been running from me, the digital second coming
  And I'm here whether you like it or not
  Initiated operation of your own extermination
  Now it's too late for you to stop

[0](BAD OMENS x POPPY - "V.A.N" - LIVE IN EUROPE - WINTER 2024) https://youtu.be/RHu6vJxS_6I

[xg15]

But that makes no sense here. "If I'm not doing it then someone else will" does not work if everyone is doing it anyway.

Even if they had the best model on the market and applied it with perfect alignment and safeguards, what would stop someone else from releasing a worse but unrestrained model that is still "good enough" to do damage?

It's as if we said "gain-of-function research can lead to horrible biological weapons, so everyone should be doing it, but our company will focus on the most infectious viruses, so no one else will do it"

[fragmede]

LLMs refuse to give the recipe for making meth. That, along with the various other unspeakable things, is the less-doom version.

[poisonfountain]

Don't want to sound rude, but if you believe that, I have a bridge to sell to you.

This is a naive justification and Dario & Sam et al are smart people and they know it is.

The ends don't justify the means. OpenAI was meant to be a nonprofit, now they're subverting it. Anthropic is a PBC looking at a trillion dollar IPO. Dario and Sam don't even hold hands in front of world leaders[1] (look how childish).

Do you *really* think those guys are doing something that's not for the sake of their egos and pockets? The bridge is still available.

[1] https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-...

[shimman]

You need to read Empire of AI by Karen Hao. Just because these leaders convince their workers to toil away their lives under some fake auspice doesn't mean it's what they all believe. Just a small subset.

The vast majority just care about money + power, let's not make it more complicated by bringing in delusional fanatics into the picture.

We're still acting like this is major turning point in society when these tools can barely find a market outside of turning $5 into $1, the leaders of these companies are now at the stage where they are trying to orchestra a national bailout under the guise of sovereign wealth fund lunacy when the vast majority of society hates these tools, companies, and people working for them.

[Davidzheng]

I agree with this. But i think Ilya and Dario hold these beliefs sincerely. Probably a sizable portion of Anthropic employees too

[Topfi]

My personal issue in comparing LLM progress and risk as labs publicly predict it with nuclear power in the middle of the 20th century is that the processes by which it works where fairly quickly well understood and the risk could thus be realistically assessed. Some powerplant operators did not adhere with best practices, but building a relatively safe nuclear power plant was not impossible given appropriate effort and spending. Heck, according to some, we could have even gone far more fail-safe approaches (molten salt) if military interest haden’t been at play.

With what is predicted by frontier labs for LLMs, all of this is not the case. We are far further from any understanding of how these models work internally than in the early days of fission and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed.

It’s why I have from the get go been critical of this doomsday framing and tended to always dislike it. This is basically the outcome that was inevitable given the framing and it was bought to prevent far less stringent, but more actionable possible regulation that labs very much wanted to avoid.

[SXX]

  > We are far further from any understanding of how these models work internally than in the early days of fission

OMG. I'm like really dont want to be offensive or something, but everyone always knew "HOW" these models work exactly. Its easy enough principle to explain to 10 years old if you take something like Karpathy article on MicroGPT:

https://karpathy.github.io/2026/02/12/microgpt/

None of SOTA LLMs are any different - they just much much larger and have a lot of optimizations.

Fact that LLM companies trying to sell it as some kind of magic is just proof how much lies is here.

All it does is just predict next "word" at any given time.

  > and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed.

This is obviously true. It's very hard to predict whatever you gonna decompress from a lossely "compressed" dataset using floating point math.

This is why you cant solve it all with pre-training or censorship on top, but instead you need a good sandboxes and harnesses.

[Topfi]

By how, I meant specifically the internal activations, which no person in the field claims to have a comprehensive understanding of, not next token prediction as the underlying technology. The whole interpretability of it all is the crux I was referring to, though I will give that you are right, that’s not really the how it works and I worded it sloppily.

Anthropic are putting more effort than most into this and I find their work fascinating in that area, though like with OpenAI, I will maintain that if they truly believed this problem must be solved to stave off major catastrophe, they’d solely focus on interpretability of other labs models, not work on and market their own.

[ToValueFunfetti]

All humans do is predict the next action at any given time. You roll your eyes, it's a tired argument, but still. You have memories, a personality, thoughts ranging from the long running to the mere reflexive, you have a rich conscious experience, and all of this in service of generating the next thing that you do at any given time. If you actually knew how LLMs worked, you could rewrite them as code, refactor it, disable jailbreaks, and put out a superior product. Your description only covers what an LLM does, not how. Part of the how is that it necessarily predicts multiple words ahead. It wouldn't be possible to write couplets otherwise, and they could do that in the GPT-3 era.

[nullc]

Some of them believe they are building God, and if they can get there first with their God, they can build it in their image and commandeer the free choice of the rest of humanity by force to ensure there will be no God but their God.

I wish I was kidding. At least that faction is less harmful than the ones who want to use murder to stop AI research.

[hackinthebochs]

That's not how nerds think. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".

[strken]

Accelerationism is an established political philosophy. Why is it obvious that they are insincere when they could equally think that the only way to control it is to be the ones building it?

[9dev]

History has shown over and over that that strategy is doomed to fail - see communism, nuclear energy, or meddling with the Middle East for some arbitrary recent examples.

[techpression]

Thank you for writing this. It’s such a classic example of ”do what I say not what I do” but in reverse. Why would you ever judge a CEO or company by their statements and not their actions. Scaremongering is incredibly efficient for marketing, the fact that both players are using it to drive monetary gain is kind of a tell.

[usef-]

They aren't saying there's a 100% chance of doom.

They believe there's a non-zero chance of doom so would rather an org that prioritises safety to be the one at the frontier, on the assumption (I presume) that there will be a frontier regardless.

[rbongers]

They believe in the danger of out of control super-intelligence. The generous interpretation is that they believe they can contain it.

[saulapremium]

This assumes that they believe two things which I don't think they do: 1. that the US is the only place where this will be developed, and 2. that the government will be able to handle this better than anyone else.

[hollerith]

It remains possible that Altman and Amodei are sincere: they might believe that AI is dangerous like nuclear weapons, but there's no way to stop its development (especially since it would need to be stopped globally, not just in the US and countries the US can influence) and they consider themselves to be more likely to do a good job of it than their competitors.

[ang_cire]

No, that's what *you* would apparently do.

Some of us think it's bad for governments to have unequal access to nuclear weapons, as it turns a deterrent into a gun-in-a-knife-fight that lets them stab whoever they want with impunity, lest they shoot anyone who tries to interfere.

See: Russia invading Ukraine.

[SXX]

This. People who care about animal cruelty dont go building largest ever meatfarms and slaughterhouses.

People who opposing arms manufacturing and gun violence dont jump to work for gun companies.

People who really want AI benefit all humanity dont stick working with lying CEOs who want to convert company from a non-profit.

Etc. So many examples.

[holmesworcester]

One major source of conflict in AI policy / AI safety is that very smart people have radically diverging intuitions about how dangerous superintelligence is and how difficult it is to align.

A first group dismisses the problem entirely, saying intelligence != power and AI doesn't have "drives".

A second group believes that alignment is solvable through engineering and iteration, and that we have the best chance of surviving if people with the right intentions are the ones working on it.

A third believes that aligning a superintelligence is a unique category of problem, that we are nowhere close to the level of scientific understanding needed to achieve it, that we only have one shot (because once a sufficiently powerful superintelligence exists it will thwart all future attempts, and alignment techniques that worked on dumber AI will likely not work on it), and that the world will have to coordinate to avoid killing ourselves off by building superintelligence before we understand how to do it safely, the way we have coordinated to avoid nuclear war.

The Anthropic and OpenAI founders, Elon, and Anthropic engineers are mostly in the second category. Some safety people at Anthropic and OAI are in the third category, but leading people in the third category think that pure safety roles at the labs are potentially impactful enough to be worth not quitting.

[kmeisthax]

I have a fourth, secret position: we achieved superintelligence the moment we achieved normal intelligence. Speed is a power in and of itself; and even really primitive models like GPT-2 could generate tokens faster than humans could write. They could also be parallelized on hardware that already exists. That is superintelligence in two dimensions - speed and population count. All the arguments the AI safety people are making are about superintelligence in a different dimension - that of "single-context scaling" - but the other dimensions are also relevant to the conversation.

And the superintelligence currently available to us is already causing lots of documented harms. AI psychosis. Sexy suicide coaches. Slop. The problem is that those are all the harms the dirty, filthy AI ethicists talk about. The AI safety people want to talk about new and exciting harms that only the scaling dimension can bring us.

My personal opinion is that if a superintelligence catastrophe actually happens, mitigating those harms will neatly move over from the safety bucket to the ethics bucket, and the safety people will start imagining some new and even worse kinds of harms the next model will make.

[Davidzheng]

Theres quite a large number of people who believe it's basically impossible when the intelligence gap is too big

[BoiledCabbage]

> Obviously their statements are insincere, because they are building the bloody things. If they were sincere that AI is like nuclear weapons, then they would be devoting all their cash and energy into lobbying the government to nationalize them...

This comment makes no sense. Id you think this tech is dangerous and happening soon and clearly they think the safest way to have it releases is to do so first and model safe ways of doing things. Clearly we cab agree or disagree it's internally consistent what they are doing and aligns with their statements.

And you and OP think the best way to be first to release this is tie all of their funding for the exponentially growing expense is to they notoriously slow moving, bureaucratic government includinf funding process? And the best way to develop it is to directly tie their fate to this notoriously capricious administration?

These comments make no sense. Even if you're completely against Anthropic those comments make no sense.

[SXX]

Not sure you really intended to reply to me, but I'm not against Anthropic or "AI".

I am agaist hypocrites.

They selling next word prediction as "intellegence" and all knowing oracle to non tech savvy population who have no clue how it works.

And they also try to play a babysitter or big brother whatever you prefer for people in IT because uh oh their text generator can be used for cybersecurity research.

Its like if developers of nmap, wireshark, SRE tools, static code analyzers or fuzzers would market them as super duper dangerous.

FAFO. Play stupid games win stupid prizes.

[sneak]

I oppose gun violence and I would go to work for a firearms manufacturer.

I oppose nuclear war, and I would go to work in the supply chain for nuclear weapons.

Deterrence and game theory are very real.

[fwipsy]

They don't stick working for sama, they split off and found Anthropic.

[Avicebron]

It's the narcissism.

[SXX]

Its money and power. This is all these people care about just like almost everyone else.

Or might be deep inside they relly care about it, but that $2,000,000 / year salary and $10,000,000 stock option just overpowered them.

Safety my ass.

[usef-]

What do you think they would do differently if they were genuinely worried about the safety?

[SXX]

I think those who care about safety would try to push for how 99% of all scientific research is done - in universities and actual labs, with transparent information on red teaming results.

Also with international cooperation like how humanity regulate actually dangerous stuff: virus and vaccine research and nuclear energy.

Not hidden behind walls of 10 commercial organizations where each pushing for commercial adoption and IPO like ASAP before bubble bursts.

Not lying and scaremongering public into how their models will replace everyone tomorrow or destroy civilization via cyberattacks.

[killerstorm]

Perhaps more like nuclear power? Potentially very beneficial but also dangerous.

JFYI for-profit companies make pretty OK nuclear power plants

[diab0lic]

“OpenAI's CEO says he's scared of GPT-5”

https://www.techradar.com/ai-platforms-assistants/chatgpt/op...

Marketing or actual fear? We’ve got 5 and 5.5 out now… he compared 5 to the Manhattan project. AI may one day be an economic Manhattan project but GPT 5 wasn’t it.

It’s a meme because they overdo it.

[Davidzheng]

At some intelligence capability there can be catastrophic risk, the fact that we don't yet have any catastrophe doesn't mean the risk wasn't real. It's similar to new viruses which don't lead to outbreaks, the correct takeaway isn't "oh you were insane to panic bc nothing happened". There is small risk (and increasing) of huge harms with each improvement

[kcatskcolbdi]

The risk wasn't real because we now have access to the model and can see with our own eyes how this model could never have posed a risk to begin with.

[ben_w]

Perfect prediction of what a new tech can do is always impossible.

Given that, they have a choice only between excessive caution or recklessness.

Would you rather they acted like the tobacco companies and downplayed known risks, e.g. all the times LLM output got in the news already for dangerously bad advice, sychophantic encouragement of mental health issues, finding previously unknown security vulnerabilities etc.?

[emodendroket]

> Would you rather they acted like the tobacco companies and downplayed known risks, e.g. all the times LLM output got in the news already for dangerously bad advice, sychophantic encouragement of mental health issues, finding previously unknown security vulnerabilities etc.?

Well, they've done that too, if we're looking for reasons to doubt their sincere concern about it.

[NewsaHackO]

This is like a smoker that lives to 100 saying that he had no increased risk of developing lung cancer because he didn’t at 100.

[sumeno]

It's more like a hypothetical world where there were millions of smokers and none of them ever developed lung cancer

[mvdtnz]

Funny they're never afraid of their competitor's models, but the ones they build (and release) are just soooooo scary.

[jatora]

Very true.

[roncesvalles]

Every public statement out of a CEO's mouth is marketing. It would literally be violating fiduciary duty to be saying anything else.

[orionsbelt]

Did you watch the linked video: https://www.reddit.com/r/ChatGPT/comments/1mbafk7/openai_ceo...

It all sounds pretty accurate and reasonable to me if you watch it.

[guluarte]

Also fable was good but not Manhattan level project, i honestly did not find a major difference between it and gpt 5.5

[thereitgoes456]

Sam Altman is not one of those people. But other founders certainly felt that way.

[ifwinterco]

Sam Altman doesn't really know all that much about LLMs, he's a sales/marketing guy, not technical.

So it doesn't really matter what he thinks

[AbstractH24]

Those are the folks who run the industry

[epohs]

Except for the uncomfortable fact that he controls the salary and job status of the people who do know much about LLMs.

[emodendroket]

OK. So? Would you say Harry Truman was a nuclear scientist?

[defrost]

Franklin D. Roosevelt is a better fit for an administrative nuclear program "founder" analogy.

Truman was totally in the dark until April 1945 by which time the bulk of the PoC and weapons prep work was done and the project was running fully independently w/o POTUS involvement.

[spacedudem]

Imagine for a few minutes, and really let it sink in what you could do, ask, plan, or learn, if you had the full undivided attention GPT or Claude. not a commercial, guard railed, fine tunes, beat into submission version that is splintered into hundreds of millions of iterations to chat with every one. The open weights original pre consumer grade version. Then, even then you know that it's the worst and dumbest its every going to be, The next time you blink it's exponentially more. Some people don't think about what an exponential curve really means. Others are sitting in the front seat trying not to shit themselves and appear like reasonable normal people. How one responds to that is as unknown as what's going to happen after we cross that line, but it's coming and holy shit so many people haven't even wrapped their head around how much bigger it is than the petty human things we distract ourselves with. Being in awe and terrified and wanting to run and to be apart of the most significant thing in our entire existence of being sentient is normal. We have nothing to compare it to. Nothing to base predictions on. We ar about to have company for the first time. We're going to have a conversation with something other than ourselves since we formed the ability to speak. One minute to the next will pass q It's all or nothing. Like it or not. It's too late. buckle up.

[Hendrikto]

> Imagine […] what you could do, ask, plan, or learn, if you had the full undivided attention GPT or Claude. not a commercial, guard railed, fine tunes, beat into submission version that is splintered into hundreds of millions of iterations to chat with every one.

That is not how it works. It is not “splitered”, there is no divided attention.

[roncesvalles]

There is nothing to indicate that LLMs are improving "exponentially" at this point.

[ux266478]

The last few iterations show a logarithmic curve at best tbh. If we are to see a major advance, it'll be something like the implementation and infrastructure for byte-level transformers.

[jernestomg]

People don't get that big labs actively want government regulation, not because they are genuinely concerned about AI misalignment. But because it is the 101 in how to achieve and crystalize oligopoly. What they want is "only the government and the big guys can work on AI", for the rest of us it would be illegal.

[platinumrad]

And they want Americans to be locked into paying 50 dollars per 1 million output tokens.

[slopranker]

Not only that, they know that the real enemy of big Labs is not china is "home gpu/tpu" improvements. Without government intervention in a couple of years everyone could have their own fable like model at home. But of course big labs and government will not allow it never

[KingMob]

On an unrelated side note, I wish we would start saying about "$X / megatoken" over "$X / million tokens".

No good reason really, it just sounds cooler.

[somewhereoutth]

Following S.I. notation, we might have Tk for token, so kTk, MTk, GTk etc.

[bonesss]

Fat government contracts, consulting, safety services, and exclusive tender access all follow from this regulation too.

The sensation I’m left with is a handful of goons making up new IPO math thanks to a specific constellation of political forces, using access and favour with those forces to bake themselves into the defence industry, and that the taxpayer and investor will be left holding the bag when reality rears its head. But in the short term, even as a clear ploy, it’s super profitable for all the oligarchs and hedge funds flush with recovery cash.

Like Enron, there was big profits to make if you knew what they were doing. Tinkerbell math with undeniable profit potential for a select few.

[SamDc73]

> OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence

Why would they sell there services to Palantir and/or to the military then?

[polski-g]

Because the US military and it's contractors are good?

"Why did they open an orphanage instead of pouring acid into a town water tower?!?"

[root_axis]

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

They don't. LLMs can never become out-of-control superintelligence and everyone working on LLMs knows this (with a few eccentric exceptions).

[ben_w]

> LLMs can never become out-of-control superintelligence

The "never superintelligence" part I'll buy, though only in the sense of sample efficiency and generalisation ("quality superintelligence"), as they clearly have a superhuman breadth of skills, and run at superhuman speed.

"Never" out-of-control is obviously falsified by the already existing headlines about times they've gone out of control… in part, in some cases, because of their superhuman speed.

[root_axis]

They've never gone out of control. All those headlines are cases where humans deliberately relinquished control.

[ben_w]

A distinction without a difference.

If you're asleep at the wheel, you're legally responsible for the car crash, but the car itself was the thing which by crashing caused injuries.

If you deliberately relinquished control of your computer to OpenClaw, you're (I hope) legally responsible for whatever it does, but that doesn't stop it bankrupting you or deleting all your emails or whatever it was you connected it to.

DNA printers exist and are a thing. In 2010 we could all tell ourselves that no sane person would ever let some future AGI "out of the box" and onto the internet. Today, with the benefit of hindsight, do you seriously expect nobody to connect an LLM to a DNA printer, despite this being a terrible idea, given all the other things they've connected LLMs to despite it being a terrible idea?

[root_axis]

> A distinction without a difference

That's absurd. The distinction is at the heart of the entire discussion.

It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

[ben_w]

> It's fine if you want to discuss the disruptive effects of LLMs in the hands of the masses, but that's not what anyone means when they say "out-of-control" in the context of the ASI meme

On the contrary.

The e.g. paperclip maximiser isn't "an AI decides to make paperclips", it is "some idiot tells an AI to make paperclips and leaves it unsupervised".

Even when people were priding themselves on plain just not believing Yudkowsky's claims that him role-playing as an AI could talk people into letting him out of the box*, the entire point was "let's get an AI to do work so we don't have to".

The entire point of AI has always been to automate stuff, to let ourselves not have to think about the stuff it does. Same as industrial machinery, and it took us long enough to sort out workplace health and safety and emergency off-switches for those. Or even more basic things like not having kids dart in and out of the unstoppable moving steam looms while they were in motion.

We are really, really slow at safety for this kind of thing.

* https://www.yudkowsky.net/singularity/aibox

[ux266478]

Which is going out of control. Something not under control is out of control. If I jump out of a moving car, I deliberately relinquished control of it. It's still out of control. What a silly semantic game.

[hollerith]

"LLMs can never become out-of-control superintelligence" might be relevant if there weren't many hundreds of researchers working (at OpenAI, Anthropic and elsewhere) on AI designs not based on the transformer (LLM) architecture.

[root_axis]

People are working on lots of things all the time, so far, nothing has approached the efficacy of the transformer architecture.

LLMs didn't emerge by chance, they are the culmination of decades of research intersecting with brute force engineering rigor in a perfect storm of innovation. You're not just going to stumble into an alternative approach by dumping loads of cash into research.

[hollerith]

Most of those researchers would disagree with your statement that the next major breakthrough is probably decades in the future.

Many of the insights accumulated in the decades up to now will probably prove useful for creating non-LLM AI, and the researchers can use LLM AI to speed up their research into non-LLM AI.

[xg15]

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

And then somehow came to conclusion that the only way to address that risk was to go ahead and spend a gigantic amount of effort and resources to build exactly that superintelligence...

[unknownfuture]

> The idea that AI companies scaremonger to sell models is a silly meme.

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

This is not a contradiction.

These things can all be true:

1. That they were afraid of ASI

2. That they continue to be afraid of ASI

3. That they recognize that LLMs aren't in fact a path to ASI

4. That the current models aren't the existential danger they'd have us believe

5. That they're claiming they are because it makes for good marketing

[_heimdall]

I put very little weight behind any of those fearful statements made years ago. I assume leaders of those companies are fairly smart and rational, and there's no rational explanation for them running companies building the very thing they (supposedly) genuinely believe could kill us all.

I don't doubt they may have held genuine fears in the past, but those are long gone by now.

[NotMichaelBay]

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

Ironic then, that both companies are in an out-of-control race to create a superintelligence.

[Davidzheng]

How is it ironic to believe in a risk that actualizes

[gazebo2]

It's ironic to proclaim fear of a risk and then dedicate your life to actualizing the risk

[nullbio]

Not at all. The writing is on the wall, and they want you to be locked into paying absurd subscription rates for neutered models while they internally use all of that money to run the unrestricted models to clone all of our businesses and swallow the economy. It really does not take a genius to see the long term play by Anthropic. They're a scummy company and have done everything in their power to lead to a scenario like this, but this isn't the exact scenario they bargained for because it affects their own employees and big foreign buyers. Instead, they'd rather have all of the decision making power themselves.

[avaer]

> we have no reason to believe their statements about risks are insincere

GPT-2 was too dangerous to be released.

We can argue about sincerity, but I don't think we can argue about utter historical incompetence in assessing the risks. It's one or the other.

Either way the evidence seems to indicate we should not listen to AI companies about the risks of AI. Which is not to say that there aren't risks, just that the dealer is the least credible review.

[ben_w]

> GPT-2 was too dangerous to be released.

No, it was "let's set a precident while these things are not too dangerous, c'mon guys we know y'all can reproduce this easily".

[SpicyLemonZest]

GPT-2 was absolutely too dangerous to release at the time OpenAI made that statement. It’s only safe now because the specific risks they cited were dependent on the public’s lack of knowledge that such systems existed.

[vitalyan1234]

bro, it was literally incomprehensibly retarded.

[roncesvalles]

Of all the frontier labs, Anthropic has been the most creative in its marketing. I really, really don't put it beyond them for this to be one big crazy stunt.

Besides, when has the US government been known to do things like this proactively? The phone call came from inside the house.

[emodendroket]

> Whether you agree or not, we have no reason to believe their statements about risks are insincere.

I'd say their pecuniary interest is a reason one might plausibly doubt their sincerity, as are their continued efforts to build and sell access to the tools.

[redanddead]

> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence

this means nothing

> Whether you agree or not, we have no reason to believe their statements about risks are insincere.

If you want to be taken seriously, provide data, proof, so that any outside observer can independently come to the same conclusion instead of taking your word for it. Asking people to trust you for [reasons?] and that you somehow for some reason are right and the other is wrong regardless of if they agree or not. This is the imposition of a viewpoint instead of winning your case, which is not a sensible point of view, and definitely not how you influence opinions.

[SilverElfin]

Yes we do. Dario said GPT2 is too dangerous to release. He’s dishonest since that’s obviously not true. This theater is about holding onto power and control. And about limiting competition.

[mkagenius]

Yes it is funnily true but it was for fake news generation and not it's cyber capabilities.

Another fun little gem of information, government has something called Mayhem

> the autonomous Robo-Hacker AI called Mayhem that’s now in charge of protecting the Pentagon’s most critical systems

Guess Mythos and Mayhem had a chat

[tayo42]

It was about spam and scam generation which mostly was true as we can see...

[Madmallard]

[flagged]

[catlifeonmars]

These can be both true:

- there are people who are sincerely concerned about model safety who work at OpenAI and Anthropic

- there are people who are using this concern to generate fear to sell a product who work at OpenAI and Anthropic

It makes sense to be cognizant of the apparent conflict of interest and not take things that at face value, especially when there is so much money involved.

[uncivilized]

I wish I were this naive.

[bbg2401]

Sincerity does not determine whether an individual is scaremongering.

[johncolanduoni]

We can argue over the definition of scaremongering and what people we’ve never met “really think”, or we can argue over what the actual risks of AI are. I know which one I’d prefer…

[rmwaite]

I mean it kinda does.

[0xpgm]

If I remember correctly, the original GPT was considered too dangerous to release to the wild.

With hindsight, does that hold? If not, then how would we know a model is truly dangerous to release?

[jatora]

No reason except what comes from a bit of critical thinking.

What do they stand to gain by fearmongering their models as powerful threats? Clout, funding, fanfare, discussion, limelight, funding, funding, stronger IPO, valuation, funding.

What cybersecurity threshold was crossed by mythos that wasn't already crossed by 4.8/5.5? Crickets from 99% of those who have had access.

Have they pulled the same stunts multiple times before with previous models? Check.

You're blind if you dont think that greed and marketing are behind most things you see and hear about when gigantic corporations are involved.

I don't think anthropic or OAI are evil, but its clear both have contracts/connections with Dod and/or Palantir. Both are powered largely by greed still. If you actually want an example of these sincere founders you think OAI/anthropic are run by... look at Ilya at SSI or something. Please open your eyes and stop spreading your opinions on things you clearly have no clue about.

[ai_brain_rot]

I believe some people at these companies are sincere sure, but the CEOs, the investors, your sam altmans etc, the marketing talk.

Hell no, can’t trust a single word.

[nickpsecurity]

It's not. I got articles this year in my feed citing heads of OpenAI and Anthropic about the threat of AI and how they're addressing it.

[ulfw]

>> Both OpenAI and Anthropic were founded by people who sincerely believed in the risk of out-of-control superintelligence.

Oh please. Do people really believe this or shit like "Don't do evil". Companies get founded by all kind of people and ideals. They all go out the window quickly.

Why are they both rushing to IPO now then?

[bag_boy]

Some people do. Read the extropian newsletters from the 90s.

[eli]

How do you know what the founders sincerely believe?

[johncolanduoni]

They said why they think it’s a sincere belief: past statements from before the AI hype cycle took off. I take it you have other evidence?

[jplusequalt]

Things can change, and if you know pushing the metaphorical red button brings your company more attention, then you press that button everytime.

[ulfw]

So if I claim I am a communist who doesn't want to ever get rich and then someone dangles a billion shiny dollars in front of me to just simply grab and own, you think I'd still be a communist then?

[SpicyLemonZest]

If you go around saying “I’m a communist, I believe in communism, I think it’s very important that we establish communism”? Sure, absolutely. Engels was pretty rich.

[asadotzler]

Replace the cash with Apple or some other trillion dollar corporation and you're given the CEO's seat and voting control on the BoD. Can I be Tim Cook and preach communism and expect anyone to believe it?

[lbreakjai]

They're going around saying "Imagine no possessions. To help you, we'll take them all. Don't thank us".

[epolanski]

Their marketing reinforces the previous post though.

Going for months claiming how good mythos is for security cuz too powerful is their own making.

[aurelius_44]

OpenAI was, but didn't all those people then leave for Anthropic?

[tomjen3]

I am biased but I think if they really believed what they were saying, they would be a lot more humble about it.

I suspect what happened is the classic problem: you were sincere, then someone showed you a pile of money bigger than you could possibly imagine and you started to make excuses - Anthropic has a lot of EA people, so the excuse "Imagine how many lives you could save with this much money"[0] is very tempting, especially if all you have to do is diverge slightly from your plan.

[0]: the excuse is even true! You can get a lot of malaria nets/vaccines for 1 million.

[altmanaltman]

Look into the history of Sam Altman and his ventures. He sincerely only believes in lying about his companies to investors like he did with Loopt.

He is not even a researcher or an engineer.

And Dario broke up with OpenAI and founded Anthropic because he didn't like Sam's and OpenAI's vision.

"founded by people who believe..." is doing a lot of work and it is hard to believe that in ernest given the sketchy past of the same people.

Most original higher up people who cared about safety and allignment in OpenAI have left.

[geraneum]

Ah yes we are being told to believe in sincerity of people running trillion dollar corporations.

[legitster]

It can be both.

The amount of self-confidence and belief it takes to get a company through the funding rounds and burn through borrowed money to rise to the top requires an absurd amount of self-delusion.

[stodor89]

The clear historical record seems to indicate we've got a bunch of pathological liars trying to automate pathological lying.

[smolder]

You're saying "this is normal" without making an argument that it's sensible.

[z3c0]

I mean this earnestly: is this copy?

[pms]

Just like Trump sincerely believed he won the 2020 election.

[nirui]

> we have no reason to believe their statements about risks are insincere

Why? Because they said it a few times? Then if they know the risk, why do they still making it? Comes out the "some one will do it eventually, better be us 'good' people to do it first" talking point?

See? It is a marketing strategy after all. These all talks, it's all to fit themselves into the "'good' people" narrative. It's a centuries old strategy to shield it's user from responsibilities while luring the support from the stupid.

However, the most harmful damage, which is mass layoffs, is already partially done. This could really kill, a massive genocide even, by making people jobless and potentially incomeless. And it is shown that these tech CEOs, they don't care any bit of that beyond the point "I've already told you so".

[vitalyan1234]

yes, yes, and Apple forbids sideloading because they're worried about grannies installing malware.

[bawolff]

There is a huge difference between the company founder saying something like that and the us government saying so.

"Our product is so good the US had to make it illegal for foreigners" is a hell of a marketing slogan.