[TechnicolorByte]

I still cannot process what’s happened to one of the most prominent and hyped companies of the past year in just one weekend.

If it’s true that Altman won’t return to OpenAI (or alternatively: that the current board won’t step down) then where does that leave OpenAI? Microsoft can’t be happy, as evidenced by reporting that Nadella was acting as mediator to bring him back. Does OpenAI survive this?

Will be super interesting when all the details come out regarding the board’s decision making. I’m especially curious how the (former) CEO of Twitch gets nominated as interim CEO.

Finally, if Altman goes his own way, it’s clear the fervent support he’s getting will lead to massive funding. Combined with the reporting that he’s trying to create his own AI chips with Middle East funding, Altman has big ambitions for being fully self reliant to own the stack completely.

No idea what the future holds for any of the players here. Reality truly is stranger than fiction.

[altdataseller]

OpenAI has hundreds more employees, all of whom are incredibly smart. While they will definitely lose the leadership and talent of those two, it’s not as if a nuclear bomb dropped on their HQ and wiped out all their engineers!

So questioning whether they will survive seems very silly and incredibly premature to me

[alsodumb]

Pretty much every researcher I know at OpenAI who are on twitter re-tweeted Sam Atlman's heart tweet with their own heart or some other supportive message.

I'm sure that's a sign that they are all team Sam - this includes a ton of researchers you see on most papers that came out of OpenAI. That's a good chunk of their research team and that'd be a very big loss. Also there are tons of engineers (and I know a few of them) who joined OpenAI recently with pure financial incentives. They'll jump to Sam's new company cause of course that's where they'd make real money.

This coupled with investors like Microsoft backing off definitely makes it fair to question the survival of OpenAI in the form we see today.

And this is exactly what makes me question Adam D'Angelo's motives as a board member. Maybe he wanted OpenAI to slow down or stop existing, to keep his Poe by Quora (and their custom assistants) relevant. GPT Agents pretty much did what Poe was doing overnight, and you can have as many as them with your existing 20$ ChatGPT Plus subscription. But who knows I'm just speculating here like everyone else.

[alex_young]

The heart tweet rebellion is about as meaningful as adding a hashtag supporting one side of your favorite conflict.

Come on. “By 5 pm everyone will quit if you don’t do x”. Response: tens of heart emojis.

[alsodumb]

It wasn't a question of "will these people quit there jobs at OpenAI and get into the job market because they support Sam".

It was a question of whether they'd leave OpenAI and join a new company that Sam starts with billions in funding at comparable or higher comp. In that case, of course who the employees are siding with matters.

[londons_explore]

Sam hasn't yet lined up the funding, so therefore they can't yet offer decent jobs, so therefore the openai employees haven't left

But they will.

[teaearlgraycold]

Talk is easy. But also the good employees will be paid well to get poached.

[hipadev23]

Anyone worth a shit will leave and go work with Sam. OpenAI will be left with a bunch of below average grifters.

[hef19898]

What is it with all this personality cult about founders, CEOs and CTOs nowadays? I thpught the cult around Steve Jobs was, bad it pales in comparison to today.

As soon as one person becomes more important than the team, as in the team starts to be structured around said person instead of with the person, that person should be replaced. Because otherwise, the team will not be functioning properly without the "star player" nor is the team more the sum of its members anymore...

[HlessClaudesman]

People love to pick sides then retroactively rationalise that decision. None of us reading about it have the facts required to make a rational judgement. So it's Johnny vs Amber time.

[Closi]

While your post sounds like something that would be true, there are loads of examples of where companies have thrived under a clear vision from a specific person.

The example of Steve Jobs used in the above post is probably a prime example - Apple just wouldn’t be the company it is today without that period of his singular vision and drive.

Of course they struggled after losing him, but the current version of Apple that has lived with Jobs and lost him is probably better than the hypothetical version of Apple where he never returned.

Great teams are important, but great teams plus great leadership is better.

[austhrow743]

In a dispute between people willing to sacrifice profit for values and those chasing the profit, why on earth would you put grifters on team values over profit?

[bertil]

I'm assuming the original comment meant that the grifters would not be extended a new offer after their colleagues learned that they were not as good as their CV said at open AI.

[throwawayhno]

Welcome to hn. Here it's all about money

[Gigablah]

Only on HN: your worth is tied to your choice of CEO.

[happytiger]

I take it you have never made a pledge to someone.

It’s a signal. The only meaning is the circumstances under which the signal is given: Sam made an ask. These were answers.

[alex_young]

This is how one answers if they actually intend to quit: https://x.com/gdb/status/1725667410387378559?s=46&t=Q5EXJgwO...

There’s nothing wrong with not following, it’s a brave and radical thing to do. A heart emoji tweet doesn’t mean much by itself.

[happytiger]

Did I say there was something wrong with either case? No. I said it was a signal. And it certainly can mean a lot by itself.

You can disagree. You can say only explicit non-emoji messages matter. That’s ok. We can agree to disagree.

[15457345234]

So is this a company or something else that starts with a c? (Thinking of a 4 letter word.)

[behringer]

Why a researcher would concern him or herself with management politics is beyond me? Particularly with a glorified sales man. Sounds like they aren't spending enough time actually working.

[bertil]

My experience of academic research is that there's a lot of energy spent on laboratory politics.

[vvrm]

Because a salesman’s skills complements those of a researcher. Salesman sells what the researcher built and brings in money to keep the lights on. Researcher gets to do what they love without having to worry about the real world. That’s a much sweeter deal than a micromanaging PI.

[wyager]

Given that the board coup was orchestrated by AI safetyists, it likely has a pretty direct bearing on life as a researcher. What are you allowed to work on? What procedures and red tape are in place? Etc.

[alsodumb]

It's not just management politics - it's about money and what they want to work on.

A lot of researchers like to work on cutting edge stuff, that actually ends up in a product. Part of the reason why so many researchers moved from Google to OpenAI was to be able to work on products that get into production.

> Particularly with a glorified sales man > Sounds like they aren't spending enough time actually working. Lmao I love how people come down to personal attacks on people.

[zq]

The two most important to OpenAI's mission - Alec Radford and Ilya Sutskever - did not respond with a heart.

[babyshake]

Presumably there is some IP assignment agreement that would make it tricky for Sam to start an OpenAI competitor without a lot of legal exposure?

[threeseed]

Team Sam = Team Money.

If you're an employee at OpenAI there is a huge opportunity to leave and get in early with decent equity at potentially the next giant tech company.

Pretty sure everyone at OpenAI's HQ in San Francisco remembers how many overnight millionaires Facebook's IPO created.

[bnralt]

There's a financial incentive. And there will be more opportunity for funding if you jump ship as well (it seems like OpenAI will have difficulty with investors after this).

But also, if you're a cutting edge researcher, do you want to stay at a company that just ousted the CEO because they thought the speed of technology was going too fast (it's sounded like this might be the reason)? You don't want to be shackled when by the organization becoming a new MIRI.

[hef19898]

It seems that MS spent 10 billion to become a minority shareholder in company controlled by a non-profit. They were warned, or maybe even Sam oversold the potential profitability of the investment.

Just as another perspective.

[zo1]

All this talk of a new venture and more money makes this smell highly fishy to me. Take this with a grain of salt, it's a random thought.

It's created huge noise and hype and controversy, and shaken things up to make people "think" they can be in on the next AI hype train "if only" they join whatever Sam Altman does now. Riding the next wave kind of thing because you have FOMO and didn't get in on the first wave.

[behringer]

If you're looking for money you probably chose wrong going with a non-profit.

[majikaja]

Money = building boring enterprise products, not building AI gods I would suspect

[threeseed]

OpenAI was building boring enterprise and developer products.

Which likely most of the company was working on.

[sangnoir]

OpenAI was building boring enterprise and developer products under Sam Altman's leadership

[j7ake]

Salaries at openai already make them millionaires.

[tempsy]

being a lowly millionaire doesn’t get you much these days. almost certainly anyone who was hired into a mid level or senior role was probably already at least a millionaire

[haldujai]

> Pretty much every researcher I know at OpenAI who are on twitter

Selection bias?

[alsodumb]

Not if it's a big sample set. There's a guy on twitter who make a list with every OpenAI researcher he could find on twitter and almost all of them did react to Sams tweet in a supportive way.

[ignoramous]

A majority of the early team that joined the non-profit OpenAI over BigTech did not do so for money but for its mission. Post-2019 hires may be more aligned with Sam but the early hires embody OpenAI's charter, Sustkever might argue.

Of course, OpenAI as a cloud-platform is DoA if Sam leaves, and that's a catastrophic business hit to take. It is a very bold decision. Whether it was a stupid one, time will tell.

[15457345234]

> every OpenAI researcher he could find on twitter

Literally the literal definition of 'selection bias' dude, like, the pure unadulterated definition of it.

[alsodumb]

Like I said, if the subset of OpenAI researchers who are on twitter is very small, sure.

But people in AI/learning community are very active on twitter. I don't know every AI researcher on OpenAIs payroll. But the fact that most active researchers (looking at the list of OpenAI paper authors, and tbh the people I know, as a researcher in this space) are on twitter.

[haldujai]

Large sample =/= (inherently) representative. What percentage of OpenAI researchers are on Twitter?

Follow-up: Why is only some fraction on Twitter?

This is almost certainly a confounder, as is often the case when discussing reactions on Twitter vs reactions in the population.

[djvdq]

They can support Sam, but still stay in the company.

[ethbr1]

How childish are employees to publicly get involved with this on Twitter?

If the CEO of my company got shitcanned and then he/she and the board were feuding?

... I'd talk to my colleagues and friends privately, and not go anywhere near the dumpster fire publicly. If I felt strongly, hell, turn in my resignation. But 100% "no comment" in public.

[djvdq]

> You should find a better place to work.

Work is work. If you start being emotional about it, it's a bad, not good, thing.

[dylan604]

These are people very active on Twitter and work for a company that unashamedly harvested all of the data it could for free with out asking to make money. It's not like shame and self-respect are allowed anywhere near this company.

[15457345234]

tl;dr: Any OAI employee tweeting about this is unhinged.

[qwertox]

Which would mean that he specifically selected who to follow due to their closeness to / alignment with Sam, pre-ousting? How would he do that?

[15457345234]

Big question!

[15457345234]

It's always been my observation that the actual heavyweights of any hardcore engineering project are the ones that avoid snarky lightweight platforms like twitter like the plague.

I would imagine that if you based hiring and firing decisions on the metric of 'how often this employee tweets' you could quite effectively cut deadwood.

With that in mind...

[alsodumb]

That's not the case with AI community. Twitter is heavily used by almost every professor/researcher/PhD student who is doing learning. Ilya has one. Heck even Jitendra Malik who's probably as old as my grand father joined twitter.

[haldujai]

Mostly for professional purposes such as networking and promoting academic activities. Sometimes for their side startups.

I rarely see a professor or PhD student voicing a political viewpoint (which is what the Sam Altman vs Ilya Sutskever debate is) on their Twitter.

[kvathupo]

Completely disagree: Yann LeCun, John Carmack, Rui Ueyama, Andrei Alexandrescu, Matt Goldbolt, Horace He, Tarun Chitra, George Hotz, etc.

[OfficialTurkey]

I have never used twitter but this strikes me as a strange take at best. Many of the most brilliant and passionate engineers I've had the pleasure to work with have been massive shitposters.

[15457345234]

> massive shitposters

Yes, agreed, but on _twitter_?

The massive_disgruntled_engineer_rant does have a lot of precedent but I've never considered twitter to be their domain. Mailing lists, maybe.

[xcv123]

Yes, on Twitter. Mailing lists are old boomer shit.

[dorkwood]

> It's always been my observation that the actual heavyweights of any hardcore engineering project are the ones that avoid snarky lightweight platforms like twitter like the plague.

What other places are there to engage with the developer community?

[15457345234]

Engagement is not necessarily constructive engagement

[dorkwood]

That's a strange thing to say. I find a lot of value in the developer community on Twitter. I wouldn't have my career without it.

I also wasn't being facetious. If there are other places to share work and ideas with developers online, I'd love to hear about them!

[karmasimida]

Discredit people using twitter is a weird take, and didn't resemble critical thinking to me.

[gardenhedge]

Since Twitter has been so controversial I don't think it's strange to discredit people using it. The people still using it are just addicted to attention.

[15457345234]

Yup. 'Tweeter' is a personality type.

[moralestapia]

Also, serious investors won't touch OpenAI with a ten foot pole after these events.

There's an idealistic bunch of people that think this was the best thing to happen to OpenAI, time will tell but I personally think this is the end of the company (and Ilya).

Satya must be quite pissed off and rightly so, he gave them big money, believed in them and got backstabbed as well; disregarding @sama, MS is their single largest investor and it didn't even warrant a courtesy phone call to let them know of all this fiasco (even thought some savants were saying they shouldn't have to, because they "only" owned 49% of the LLC. LMAO).

Next bit of news will be Microsoft pulling out of the deal but, unlike this board, Satya is not a manchild going through a crisis, so it will happen without it being a scandal. MS should probably just grow their own AI in-house at this point, they have all the resources in the world to do so. People who think that MS (a ~50 old company, with 200k employees, valued at almost 3 trillion) is now lost without OpenAI and the Ilya gang must have room temperature IQs.

[visarga]

200k MS employees can't do what 500 from OAI can, the more you pile on the problem, the worse the outcome. The problem with Microsoft is that, like Google, Amazon and IBM, they are not a good medium for radical innovation, are old, ossified companies. Apple used to be nimble when Steve was alive, but went to coasting mode since then. Having large revenue from old business is an obstacle in the new world, maybe Apple was nimble because it had small market share.

[codebolt]

MS isn't starting from scratch, it already has the weights of the worlds most powerful LM, and it's all running on their datacenters. Even without Sam, they just need to keep the current momentum going. Maybe axe ChatGPT and focus solely on Bing/Copilot going forward. It would give me great satisfaction to see the laughing stock search engine of the past decade being the undisputed face of AI over the next.

[hn_throwaway_99]

> Apple used to be nimble when Steve was alive, but went to coasting mode since then

Give me a break. Apple Watch and Air pods are far and away leaders in their category, Apple's silicon is a huge leap forward, there is innovation in displays, CarPlay is the standard auto interface for millions of people, while I may question the utility the Vision Pro is a technological marvel, iPhone is still a juggernaut (and the only one of these examples that predate Jobs' passing), etc. etc.

Other companies dream about "coasting" as successfully.

[Freedom2]

> Apple Watch and Air pods are far and away leaders in their category,

By what metric? I prefer open hardware and modifiable software - these products are in no way leaders for me. Not to mention all the bluetooth issues my family and friends have had when trying to use them.

[cloverich]

My first question to this scenario would be: Could MS provide the seed funding for Sam's next gig? As in, they bet on OpenAI, and either OpenAI keeps on keeping on or Sam's gig steals the thunder, and they presumably have the cash to play a role in both.

[moralestapia]

Surprise surprise!

https://x.com/satyanadella/status/1726509045803336122?s=46

[didibus]

But OpenAI is a non for profit that was exploring a goal that it saw financial incentives as misaligned.

It's what kind of got it achieved. Because every other company didn't really see the benefit of going straight to AGI, instead working on incremental addition and small iteration.

I don't know why the board decided to do what it did, but maybe it sees that OpenAI was moving away from R&D and too much into operations and selling a product.

So my point is that, OpenAI started as a charity and literally was setup in a way to protect that model, by having the for-profit arm be governed by the non-for-profit wing.

The funny thing is, Sam Altman himself was part of the people who wanted it that way, along with Elon Musk, Illya and others.

And I kind of agree, what kind of future is there here? OoenAI becomes another billion dollar startup that what? Eventually sells out with a big exit?

It's possible to see the whole venture as taking away from the goal set out by the non for profit.

[karmasimida]

Survive as existing? They will.

But this is a disaster that can't be sugarcoated. Working in an AI company with a doomer as head is ridiculous. It will be like working in a tobacco company advocating for lung cancer awareness.

I don't think the new CEO can do anything to get back trust in record short amount of time. The sam loyalists will leave. The question remain, how is the new CEO going to hire new people, and will he be able to do so fast enough, and the ones who remain will accept the company that is a drastically different.

[peanuty1]

Surely the employees knew before joining that OpenAI is a non-profit aiming to develop safe AGI?

[alexgartrell]

OpenAI's recruiting pitch was 5-10+ million/year in the form of equity. The structure of the grants is super weird by traditional big-company standards, but it was plausible enough that you could squint and call it the same. I'd posit that many of the people jumping to OpenAI are doing it for the cash and not the mission.

https://the-decoder.com/openai-lures-googles-top-ai-research....

[sgift]

They thought so. Now, they know that instead they work for one aiming to satisfy the ego of a specific group of people - same as everywhere else.

[bottlepalm]

Ah yes you're either a doomer or e/acc. Pick an extreme. Everything must be polarized.

[astrange]

There's a character in HPMOR named after the new CEO.

(That's the religious text of the anti-AI cult that founded OpenAI. It's in the form of a very long Harry Potter fanfic.)

[tempusalaria]

Imagine how bad a reputation EA would have if the general public knew about HPMOR

[xvector]

Even HP fanfiction lovers HATED HPMOR. It had a clowny reputation

It is wild to see how closely connected the web is though. Yudkowsky, Shear, and Sutskever. The EA movement today controls a staggering amount of power.

[FeepingCreature]

Sorry, which character are you talking about? (Also lol "religious text", how dare people have didactic opinions.)

[astrange]

The one with the same name as the new CEO. Pretty straightforward.

> Also lol "religious text", how dare people have didactic opinions.

That's not what a religious text is, that'd just be a blog post. It's the part where reading it causes you to join a cult group house polycule and donate all your money to stopping computers from becoming alive.

[whatshisface]

Is Chat GPT writing this whole dialogue?

[questime]

The perception right now is that the board doesn't care about investors, this will kill this company that is burning money at an insane rate. Employees will run for the exits unless they are convinced that there is a future exit.

[thekevan]

Funny you should reference a nuclear bomb. This was 14 minutes after your post.

https://twitter.com/karpathy/status/1726478716166123851

[jxf]

But a number of those other employees have said they'll leave if Altman isn't rehired.

[zombiwoof]

Bullshit. They are not quitting

[TechnicolorByte]

Even if you don’t believe many employees would consider leaving for Altman, I find it probable that many would consider leaving for financial reasons. What will their PPUs be worth if OpenAI is seen as a funding risk?

[bartimus]

Maybe not instantly. But there's a version where they don't agree with certain decisions and will now be more open to other opportunities.

[15457345234]

They're either not quitting or they've outed themselves as being part of a personality cult and they'll just hinder things if they're not ejected promptly.

[icy_deadposts]

You're right. They're fired.

[spaceman_2020]

If the funding dries up for OpenAI, those engineers have no incentive to keep working there. No point wasting your career on an organization that's destined to die.

[laurels-marts]

> and talent of those two

You are aware that more than just 2 people departed?

[pedrosorio]

The GPT-4 pre-training research lead quit on Friday.

[patapong]

I am guessing they are super reliant on Microsoft to keep running ChatGPT... If Microsoft decides to get out and finds a way they would be in deep trouble.

[sangnoir]

I'm sure Google will throw a couple of billions their way, given the chance

[exitb]

Why though? Companies invest to see profit or get products they can sell. This is not only about the CEO. The CEO change signals a radical strategic shift.

[blast]

> it’s not as if a nuclear bomb dropped on their HQ

Oh yes it is.

[DantesKite]

Andrej Karpathy literally just tweeted the nuclear radiation emoji lol.

[simultsop]

With a PR damage such this one, if they survive it will be a miracle.

[chubot]

What I think is funny is how the whole "we're just doing this to make sure AI is safe" meme breaks down, if you have OpenAI, Anthropic, and Altman AI all competing, which seems likely now.

Do you really need all 3? Is each one going to claim that they're the only ones who can develop AGI safely?

Since Sam left, now OpenAI is unsafe? But I thought they were the safe ones, and he was being reckless.

Or is Sam just going to abandon the pretense, competing Google- and Microsoft-style? e.g. doing placement deals, attracting eyeballs, and crushing the competition.

Surely that's what you need for safety?

[ryanSrich]

Can someone explain to me what they mean by "safe" AGI? I've looked in many places and everyone is extremely vague. Certainly no one is suggesting these systems can become "alive", so what exactly are we trying to remain safe from? Job loss?

[cthalupa]

>Certainly no one is suggesting these systems can become "alive"

No, that very much is the fear. They believe that by training AI on all of the things that it takes to make AI, at a certain level of sophistication, the AI can rapidly and continually improve itself until it becomes a superintelligence.

[ryanSrich]

That's not alive in any meaningful sense.

When I say alive, I mean it's like something to be that thing. The lights are on. It has subjective experience.

It seems many are defining ASI as just a really fast self learning computer. And while sure, given the wrong type of access and motive, that could be dangerous. But it isn't anymore dangerous than any other faulty software that has access to sensitive systems.

[FeepingCreature]

You're thinking about "alive" as "humanlike" as "subjective experience" as "dangerous". Instead, think of agentic behavior as a certain kind of algorithm. You don't need the human cognitive architecture to execute an input/output loop trying to maximize the value of a certain function over states of reality.

> But it isn't anymore dangerous than any other faulty software that has access to sensitive systems.

Seems to me that can be unboundedly dangerous? Like, I don't see you making an argument here that there's a limit to what kind of dangerous that class entails.

[reducesuffering]

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

Signed by Sam Altman, Ilya Sutskever, Yoshua Bengio, Geoff Hinton, Demis Hassabis (DeepMind CEO), Dario Amodei (Anthropic CEO), and Bill Gates.

https://twitter.com/robbensinger/status/1726039794197872939

[dragonwriter]

> Certainly no one is suggesting these systems can become "alive",

Lots of people have been publicly suggesting that, and that, if not properly aligned, it poses an existential risk to human civilization; that group includes pretty much the entire founding team of OpenAI, including Altman.

The perception of that risk as the downside, as well as the perception that on the other side there is the promise of almost unlimited upside for humanity from properly aligned AI, is pretty much the entire motivation for the OpenAI nonprofit.

[idontwantthis]

How does it actually kill a person? When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire?

[dragonwriter]

> When does it stop existing in boxes that require a continuous source of electricity and can’t survive water or fire?

When someone runs a model in a reasonably durable housing with a battery?

(I'm not big on the AI as destroyer or saviour cult myself, but that particular question doesn't seem like all that big of a refutation of it.)

[idontwantthis]

But my point is what is it actually doing to reach out and touch someone in the doomsday scenario?

[grey-area]

The network is the computer.

If you live in a city right now there are millions of networked computers that humans depend on in their everyday life and do not want to turn off. Many of those computers keep humans alive (grid control, traffic control, comms, hospitals etc). Some are actual robotic killing machines but most have other purposes. Hardly any are air-gapped nowadays and all our security assumes the network nodes have no agency.

A super intelligence residing in that network would be very difficult to kill and could very easily kill lots of people (destroy a dam for example), however that sort of crude threat is unlikely to be a problem. There are lots of potentially bad scenarios though many of them involving the wrong sort of dictator getting control of such an intelligence. There are legitimate concerns here IMO.

[upwardbound]

One route is if AI (not through malice but simply through incompetence) plays a part in a terrorist plan to trick the US and China or US and Russia into fighting an unwanted nuclear war. A working group I’m a part of, DISARM:SIMC4, has a lot of papers about this here: https://simc4.org

[hurryer]

Since you work on this, do you think leaders will wait until confirmation of actual nuclear detonations, maybe on TV, before believing that a massive attack was launched?

[justcool393]

so the plot of WarGames?

[mlindner]

What does "properly aligned" even mean? Democracies even with countries don't have alignment, let alone democracies across the world. They're a complete mess of many conflicting and contradictory stances and opinions.

This sounds, to me, like the company leadership want the ability to do some sort of picking of winners and losers, bypassing the electorate.

[krisoft]

> What does "properly aligned" even mean?

You know those stories where someone makes a pact with the devil/djin/other wish granting entity, and the entity does one interpretation of what was wished, but since it is not what the wisher intended it all goes terribly wrong? The idea of alignment is to make the djin which not only can grant wishes, but it does them according to the unstated intention of the wisher.

You might have heard the story of the paper clip maximiser. The leadership of the paperclip factory buys one of those fancy new AI agents and asks it to maximise paperclip production.

What a not-well aligned AI might do: Reach out through the internet to a drug cartel’s communication nodes. Hack the communications and take over the operation. Optimise the drug traficking operations to gain more profit. Divert the funds to manufacture weapons for multiple competing factions on multiple crisis points on Earth. Use the factions against each other. Divert the funds and the weapons to protect a rapidly expanding paperclip factory. Manipulate and blackmail world leaders into inaction. If the original leaders of the paperclip factory try to stop the AI eliminate them, since that is the way to maximise paper clip production. And this is just the begining.

What a well alligned AI would do: Fine tune the paperclip manufacturing machinery to eliminate rejects. Reorganise the factory layout to optimise logistics. Run a succesfull advertising campaign which leads to a 130% increase in sales. (Because clearly this is what the factory owner intended it to do. Altough they did a poor job of expressing their wishes.)

[mlindner]

I like your extremist example, however I fear what "properly aligned" means for more vague situations, where it is not at all clear what the "correct" path is, or worse, that it's very clear what "correct" is for some people, but that "correct" is another man's "evil".

[upwardbound]

Any AGI must at a minimum be aligned with these two values:

(1) humanity should not be subjugated

(2) humanity should not go extinct before it’s our time

Even Kim Jong Un would agree with these principles.

Currently, any AGI or ASI built based on any of the known architectures contemplated in the literature which have been invented thus far would not meet a beyond-a-reasonable-doubt standard of being aligned with these two values.

[lwhi]

I think this is a crazy set of values.

'.. before it's our time' is definitely in the eye of the beholder.

[bartimus]

It being "alive" is sort of what AGI implies (depending on your definition of life).

Now consider the training has caused it to have undesirable behavior (misaligned with human values).

[hsrada]

Death.

The default consequence of AGI's arrival is doom. Aligning a super intelligence with our desires is a problem that no one has solved yet.

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

----

Listen to Dwarkesh Podcast with Eliezer or Carl Shulman to know more about this.

[ryanSrich]

I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work.

I'm not suggesting we don't see ASI in some distant future, maybe 100+ years away. But to suggest we're even within a decade of having ASI seems silly to me. Maybe there's research I haven't read, but as a daily user of AI, it's hilarious to think people are existentially concerned with it.

[upwardbound]

> maybe 100+ years away

I have two toddlers. This is within their lifetimes no matter what. I think about this every day because it affects them directly. Some of the bad outcomes of ASI involve what’s called s-risk (“suffering risk”) which is the class of outcomes like the one depicted in The Matrix where humans do not go extinct but are subjugated and suffer. I will do anything to prevent that from happening to my children.

[FeepingCreature]

> I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work.

Maybe they don't seem that to others? I mean, you're not really making an argument here. I also use GPT daily and I'm definitely worried. It seems to me that we're pretty close to a point where a system using GPT as a strategy generator can "close the loop" and generate its own training data on a short timeframe. At that point, all bets are off.

[hsrada]

> I like science fiction too, but all of these potential scenarios seem so far removed from the low level realities of how these systems work.

Today, yes. Nobody is saying GPT-3 or 4 or even 5 will cause this. None of the chatbots we have today will evolve to be the AGI that everyone is fearing.

But when you go beyond that, it becomes difficult to ignore trend lines.

Here's a detailed scenario breakdown of how it might come to be –https://www.dwarkeshpatel.com/p/carl-shulman

[bnralt]

> Aligning a super intelligence with our desires is a problem that no one has solved yet.

It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions.

[ethbr1]

No, the problem with AGI is potential exponential growth.

So less like an alien invasion.

And more like a pandemic at the speed of light.

[mlyle]

That's assuming a big overshoot of human intelligence and goal-seeking. An average human capability counts as "AGI."

If lots of the smartest human minds make AGI, and it exceeds a mediocre human-- why assume it can make itself more efficient or bigger? Indeed, even if it's smarter than the collective effort of the scientists that made it, there's no real guarantee that there's lots of low hanging fruit for it to self-improve.

I think the near problem with AGI isn't a potential tech singularity, but instead just the tendency for it potentially to be societally destabilizing.

[astrange]

Exponential growth is not intrinsically a feature of an AGI except that you've decided it is. It's also almost certainly impossible.

Main problems stopping it are:

- no intelligent agent is motivated to improve itself because the new improved thing would be someone else, and not it.

- that costs money and you're just pretending everything is free.

[dminik]

We see alignment problems all the time. Current systems are not particularly smart or dangerous. But they lie on purpose and funnily enough considering the current situation, Microsoft's attempt was threatening users shortly after launch.

[MrScruff]

The argument would be that by the time we see the problem it will be too late. We didn’t really anticipate the unreasonable effectiveness of transformers until people started scaling them, which happened very quickly.

[hurryer]

Survivorship bias.

It's like saying don't worry about global thermonuclear war because we haven't seen it yet.

The Neandethals on the other hand have encountered a super-intelligence.

[FeepingCreature]

> It's a problem that we haven't seen the existence of yet. It's like saying no one has solved the problem of alien invasions.

But if we're seeing the existence of an unaligned superintelligence, surely it's squarely too late to do something about it.

[jbgt]

Perhaps listen to this podcast instead.

https://www.matthewgeleta.com/p/joscha-bach-ai-risk-and-the-...

[jazzyjackson]

I'm not sure that it's a matter of "knowing" as much as it is "believing"

[ilrwbwrkhv]

There is absolutely no AGI risk. These are mere marketing ploys to sell a chatbot / feel super important. A fancy chatbot, but a chatbot none the less.

[cornel_io]

Smart people like Ilya really are worried about extinction, not piddling near-term stuff like job loss or some chat app saying some stuff that will hurt someone's feelings.

The worry is not necessarily that the systems become "alive", though, we are already bad enough ourselves as a species in terms of motivation so machines don't need to supply the murderous intent: at any given moment there are at least thousands if not millions of people on the planet that would love nothing more than be able to push a button an murder millions of other people in some outgroup. That's very obvious if you pay even a little bit of attention to any of the Israel/Palestine hatred going back and forth lately. [There are probably at least hundreds to thousands that are insane enough to want to destroy all of humanity if they could, for that matter...] If AI becomes powerful enough to make it easy for a small group to kill large numbers of people that they hate, we are probably all going to end up dead, because almost all of us belong to a group that someone wants to exterminate.

Killing people isn't a super difficult problem, so I don't think you really even need AGI to get to that sort of an outcome, TBH, which is why I think a lot of the worry is misplaced. I think the sort of control systems that we could pretty easily build with the LLMs of today could very competently execute genocides if they were paired with suitably advanced robotics, it's the latter that is lacking. But in any case, the concern is that having even stronger AI, especially once it reliably surpasses us in every way, makes it even easier to imagine an effectively unstoppable extermination campaign that runs on its own and couldn't be stopped even by the people who started it up.

I personally think that stronger AI is also the solution and we're already too far down the cat-and-mouse rabbithole to pause the game (which some e/acc people believe as the main reason they want to push forward faster and make sure a good AI is the first one to really achieve full domination), but that's a different discussion.

[huytersd]

They give it stupid terms like “alignment” to make it opaque to the common person. It’s basically sitting on your hands and pointing to sci-fi as to why progress should be stopped.

[FeepingCreature]

This is why the superior term is "AI notkilleveryoneism."

[zombiwoof]

Two words: Laundry Buddy

Sam doomed himself. Laundry Buddy is the new Clippy

[chubot]

If we do not release Laundry Buddy, that increases humanity's extinction risk

[0xDEAFBEAD]

>Do you really need all 3? Is each one going to claim that they're the only ones who can develop AGI safely?

What we need at this point is a neutral 3rd party who can examine their safety claims in detail and give a relatively objective report to the public.

[asdkl234890]

And as if researches in other nations like China will sit on their hands and do nothing. They are busy catching up but without any ethics boards.

[tempsy]

Yeah Emmett Shear seems like an odd choice if they’re worried about retention because 1) Twitch was never known to be a particularly great place to work and 2) he stepped down for some reason and not because Twitch was in an amazing place or anything at the time

[himaraya]

Emmett's just a placeholder after Murati turned. I suspect he won't stay in his position for long.

[PepperdineG]

Recursive Interim CEOs. Will there be a Mandelbrot set of Interim CEOs?

[mickdarling]

This particular board won’t even let ChatGPT help the CEO because they’re afraid there’s a Basilisk hiding in every response.

[sgift]

The new research focus demanded by the board in the name of safety will be an CEO AI, which will be aligned to humanities interests - the benchmark to show this will be if it does whatever the board wants. It's the only way to make sure they cannot be stabbed in the back again by a pesky human.

[dehrmann]

And people say there aren't real-world applications to n-ary tree rebalancing.

[vintermann]

I laughed, but actually I think the utility of tree rebalancing is widely appreciated!

[sanxiyn]

Emmett Shear is probably the person most friendly to OpenAI board's AI safety agenda among possible candidates. Source: have a look at his Twitter.

[theptip]

The big question in my mind is the reported threat from MSFT to withhold cloud credits (i.e. the actual currency of their $10B investment). Is this true? And are they going to follow through?

I don't buy for a second that enough employees will walk to sink the company (though it could be very be disruptive). But for OpenAI, losing a big chunk of their compute could mean they are unable to support their userbase and that could permanently damage their market position.

[justcool393]

was it even reported? i heard a bunch of stuff that seemed to be hypothetical guessing like "satya must be furious" that seemed to morph into "it was reported satya is furious"

i've seen similar with the cloud credits thing, people just pontificating whether it's even a viable strategy.

[cthalupa]

The report was that investors were talking to microsoft about the threat to withhold credits.

Which does not say whether microsoft was open to the idea or ultimately chose to pursue that path.

[borissk]

MS is not going to randomly withhold cloud credits, as OpenAI is going to sue them for billions of damages.

[vaxman]

So they should keep buying H100s (and H200s) and pouring billions into their own chips on the expectation that OpenAI will fulfill its contractual obligations under THESE circumstances? If they stop doing that, how long before all of Azure is busy on a money losing chat program under all new leadership that doesn’t have the same plan that was sold to MSFT?

[ignoramous]

> No idea what the future holds for any of the players here. Reality truly is stranger than fiction.

Is it though? "No outcome where [OpenAI] is one of the big five technology companies. My hope is that we can do a lot more good for the world than just become another corporation that gets that big." -Adam D'Angelo

[Palmik]

I guess he would prefer is the existing incumbents got even larger, or if his competitor to ChatGPT (Poe) could capture significant fraction of the market.

[_factor]

Can’t beat em so join em? You’re framing this as a capitalist competition. Non-profits don’t care if their “competitors” win market share.

[jumelles]

The dirty secret of the business world is that the C-suite is the most easily replaceable.

[mym1990]

Middle East funding and fully self reliant seem to be at odds here.

[dehrmann]

> I still cannot process what’s happened to one of the most prominent and hyped companies of the past year in just one weekend.

That's kinda what happened. The latest gist I read was that the non-profit, idealistic(?) board clashed with the for-profit, hypergrowth CEO over the direction to take the company. When you read the board's bios, they're weren't ready for this job (few are; these rocket ship stories are rare), the rocket ship got ahead of their non-profit goals, and they found themselves in over their heads, then failed to game out how this would go over (poor communication with MS, not expecting Altman to get so much support).

From here, the remaining board either needs to either surface some very damning evidence (the memo ain't it) or step down and let MS and Sequoia find a new board (even if they're not officially entitled to do that). Someone needs to be saying mea culpa.

[sanxiyn]

I am not sure why. As far as I can tell, the board doesn't need to answer to anyone.

[g42gregory]

Unfortunately (or fortunately?), you always have to answer to somebody. In this board's case, they have to answer to investors, Microsoft in particular. Why? Because Microsoft can pull the money (apparently they only sent a fraction of $10Bn so far) and can sabotage the partnership deal. The OpenAI won't meet the payroll and won't be able to run the GPU farm. Microsoft already threatened to do exactly that.

My suspicion is that Microsoft will do exactly that: they will pull the money, sabotage the partnership deal and focus on rebuilding GPT in-house (with some of the key OpenAI people hired away). They will do this gradually, on their own timetable, so that it does not disrupt the GPT Azure access to their own customers.

I doubt that there could be a replacement for the Microsoft deal, because who would want to go through this again? OpenAI might be able to raise a billion or two from the hard core AI Safety enthusiasts, but they won't be able to raise $10s of Billions needed to run the next cycle of scaling.

[dehrmann]

I think the diagram I saw showed they don't actually answer to MS, VCs, or employee investors? And not even that they're out-voted, they don't answer to them at all.

[sanxiyn]

As far as I can tell, this is correct.

[yeck]

Well, despite what Musk did, X (Twitter?) has still been limping along for quite a while now. While more abrupt and surprising, this doesn't seem nearly as bad as that.

[extheat]

This is far worse. OpenAI simply cannot survive without Microsoft and skeleton staff. It's not like a static codebase where you can keep the service up and running indefinitely barring bugs. Why would anyone building with the OpenAI APIs, their customers, have any faith in the company if they openly don't care about business? Working on AI is highly capital intensive, on the scale of many tens of billions of dollars. Where are they going to get that funding? How will they pay their staff? There is no way Microsoft is going to HODL after this embarrassment.

[yeck]

Musk fired most of the engineers. I'd be pretty surprised if we see the level of attrition at OpenAI getting within an order of magnitude of that. We are just making predictions, though. I could be way off the mark and many more people are willing to jump ship than I imagine.

As for Microsoft, if they let OpenAI go, then what? Does Google pick them up? Elon? They are still looking to invent AGI, so I'd be surprised if no one wants to take advantage of that opportunity. I'd expect Microsoft to be aware of this and weigh into their calculus.

[seydor]

Can microsoft buy IP from openAI? recruit their engineers? asking for a friend

[BOOSTERHIDROGEN]

Exclusive use up until pre-AGI tech.

[brotchie]

Don't fully believe this, but the only rational explanation I can see is that Ilya knows they have AGI.

   - Nuke employee morale: massive attrition, not getting upside (tender offer),
   - Nuke the talent magnet: who's going to want to work there now?
   - Nuke Microsoft relationship: all those GPUs gone,
   - Nuke future fundraising: who's going to fund this shit show?

Just doesn't make sense.

[vultour]

People really need to stop with this AGI bullshit. They make a glorified Markov chain and suddenly they should have AGI? Self-driving cars are barely able to stay on the road after all this time, but sure, someone's hiding conscious machines in their basement.

[lazystar]

burnout and sleep deprivation can lead to some pretty bad choices; thats why you want to surround yourself with people that will stand up to you when your ideas and plans suffer from too much tunnel vision. sounds like the other 3 board members were yes-men/women; the house of cards was there for a while, it seems.

[upwardbound]

Reminds me of the very ending of the show Silicon Valley. Crazy twist and great last two episodes of the show.

[GreedClarifies]

Don’t have twits on the board. Lesson learnt.

[vaxman]

No, OpenAI will not survive as a company with more than one shareholder. At the end of the day, MSFT has a fiduciary duty to its own shareholders. MSFT has set certain expectations for its own financial performance based on its agreements with OpenAI and MSFT shares traded based on those expectations. Now OpenAI has sustained a hemorrhage of its leadership that negotiated those agreements, including a public admission by OpenAI of deception in their boardroom and private talk of a potential competitor involving employees. The only question is if OpenAI will capitulate or the lawyers and supply chain will be leveraged to compel their cooperation with protecting the MSFT shareholders. MSFT has deep enough pockets to retain all of the workers. One way or another, the IP and their ops are now the property of the bank, in this case MSFT shareholders. Let’s hope nobody goes to jail by resisting what is a standard cleanup operation at this point.

[vaxman]

“Sorry, we are reporting a write down of $10 billion due to potential misrepresentations of commercial intent that occurred in our OpenAI portfolio.”

Things you will never hear Satya Nadella say. Way more likely he will coordinate to unify as much of their workers as he can to continue on as a subsidiary, with the rest left to go work something out with other players crazy/desperate enough to trust them.