[SXX]

This. People who care about animal cruelty dont go building largest ever meatfarms and slaughterhouses.

People who opposing arms manufacturing and gun violence dont jump to work for gun companies.

People who really want AI benefit all humanity dont stick working with lying CEOs who want to convert company from a non-profit.

Etc. So many examples.

[holmesworcester]

One major source of conflict in AI policy / AI safety is that very smart people have radically diverging intuitions about how dangerous superintelligence is and how difficult it is to align.

A first group dismisses the problem entirely, saying intelligence != power and AI doesn't have "drives".

A second group believes that alignment is solvable through engineering and iteration, and that we have the best chance of surviving if people with the right intentions are the ones working on it.

A third believes that aligning a superintelligence is a unique category of problem, that we are nowhere close to the level of scientific understanding needed to achieve it, that we only have one shot (because once a sufficiently powerful superintelligence exists it will thwart all future attempts, and alignment techniques that worked on dumber AI will likely not work on it), and that the world will have to coordinate to avoid killing ourselves off by building superintelligence before we understand how to do it safely, the way we have coordinated to avoid nuclear war.

The Anthropic and OpenAI founders, Elon, and Anthropic engineers are mostly in the second category. Some safety people at Anthropic and OAI are in the third category, but leading people in the third category think that pure safety roles at the labs are potentially impactful enough to be worth not quitting.

[kmeisthax]

I have a fourth, secret position: we achieved superintelligence the moment we achieved normal intelligence. Speed is a power in and of itself; and even really primitive models like GPT-2 could generate tokens faster than humans could write. They could also be parallelized on hardware that already exists. That is superintelligence in two dimensions - speed and population count. All the arguments the AI safety people are making are about superintelligence in a different dimension - that of "single-context scaling" - but the other dimensions are also relevant to the conversation.

And the superintelligence currently available to us is already causing lots of documented harms. AI psychosis. Sexy suicide coaches. Slop. The problem is that those are all the harms the dirty, filthy AI ethicists talk about. The AI safety people want to talk about new and exciting harms that only the scaling dimension can bring us.

My personal opinion is that if a superintelligence catastrophe actually happens, mitigating those harms will neatly move over from the safety bucket to the ethics bucket, and the safety people will start imagining some new and even worse kinds of harms the next model will make.

[Davidzheng]

Theres quite a large number of people who believe it's basically impossible when the intelligence gap is too big

[BoiledCabbage]

> Obviously their statements are insincere, because they are building the bloody things. If they were sincere that AI is like nuclear weapons, then they would be devoting all their cash and energy into lobbying the government to nationalize them...

This comment makes no sense. Id you think this tech is dangerous and happening soon and clearly they think the safest way to have it releases is to do so first and model safe ways of doing things. Clearly we cab agree or disagree it's internally consistent what they are doing and aligns with their statements.

And you and OP think the best way to be first to release this is tie all of their funding for the exponentially growing expense is to they notoriously slow moving, bureaucratic government includinf funding process? And the best way to develop it is to directly tie their fate to this notoriously capricious administration?

These comments make no sense. Even if you're completely against Anthropic those comments make no sense.

[SXX]

Not sure you really intended to reply to me, but I'm not against Anthropic or "AI".

I am agaist hypocrites.

They selling next word prediction as "intellegence" and all knowing oracle to non tech savvy population who have no clue how it works.

And they also try to play a babysitter or big brother whatever you prefer for people in IT because uh oh their text generator can be used for cybersecurity research.

Its like if developers of nmap, wireshark, SRE tools, static code analyzers or fuzzers would market them as super duper dangerous.

FAFO. Play stupid games win stupid prizes.

[sneak]

I oppose gun violence and I would go to work for a firearms manufacturer.

I oppose nuclear war, and I would go to work in the supply chain for nuclear weapons.

Deterrence and game theory are very real.

[fwipsy]

They don't stick working for sama, they split off and found Anthropic.

[Avicebron]

It's the narcissism.

[SXX]

Its money and power. This is all these people care about just like almost everyone else.

Or might be deep inside they relly care about it, but that $2,000,000 / year salary and $10,000,000 stock option just overpowered them.

Safety my ass.

[usef-]

What do you think they would do differently if they were genuinely worried about the safety?

[SXX]

I think those who care about safety would try to push for how 99% of all scientific research is done - in universities and actual labs, with transparent information on red teaming results.

Also with international cooperation like how humanity regulate actually dangerous stuff: virus and vaccine research and nuclear energy.

Not hidden behind walls of 10 commercial organizations where each pushing for commercial adoption and IPO like ASAP before bubble bursts.

Not lying and scaremongering public into how their models will replace everyone tomorrow or destroy civilization via cyberattacks.

[usef-]

That line of thinking (public goods) is why the same people started OpenAI as a non profit originally.

Notice how almost no Universities are producing large models?

A key problem is that orgs can't get enough funds to stay on the frontier. And they believe they must be on the frontier to do (and apply) safety research. OpenAI needed to spin off a for-profit subsidiary to accept investment to build things.

And it seem(ed) hard to get one government to fund and take safety seriously, let alone an international cooperation.

If they started a university/gov cooperative to solve this, do you think they would do less of the "scaremongering of the public" talk? My guess it that it would be similar.

The same kind of restrictions that you hint at (eg, treating it like a public virus research) are why they rub a lot of people the wrong way in the corporate world, I think. Normally companies downplay the risks of their own products. See cigarette companies. Anthropic do still publish safety research and red teaming info. But I do think they honestly believe they can't do this work without the resources of a company, and they were burnt by the non-profit structure (Anthropic has a "Long-Term Benefit Trust" instead).

We should definitely keep them to account, but I don't personally think Anthropic have acted in a way broadly inconsistent with safety belief yet. Many of these decisions are self-serving too (eg. protecting models) so they also haven't been seriously tested, either. But the individuals do have a very long history of talking about it (including hurting their own reputation) from even before the chatpgt-moment money train rolled in.

edit: for clarity, but still messily/quickly written