[mrshadowgoose]

I fully agree that malicious corporations and governments are the largest risk here. However, I think it's incredibly important to reject the reframing of "AI safety" as anything other than the existential risk AGI poses to most of humanity.

What will the world look like when AGI is finally achieved, and the corporations and governments that control them rapidly have millions of useless mouths to feed? We might end up living in a utopic post-scarcity society where literally every basic need is furnished by a fully automated industrial base. But there are no guarantees that the entities in control will take things in that direction.

AI safety is not about whether "tech bros are going to be mean to women". AI safety is about whether my government is concerned with my continued comfortable existence once my economic value as a general intelligence is reduced to zero.

[somenameforme]

The risk in your scenario there is not really coming from AGI, but how selective access to such might enable people to harm others. If you have access to AGI capable of enabling you to realistically build out your utopian vision, then it matters not what other groups are doing - short of coming to try to actively destroy you. You'd have millions that would join you, and could turn that vision into a reality regardless of whatever currently powerful entities think about it. So the real danger does not really seem to be AGI, but the restricted access to such.

The focus on "safety" all but guarantees that there are going to be two sets of "AGI", if such is ever to be achieved. There will be lobotomized, censored, and politically obedient version that the public has access to. And then there will be the "real" system that militaries, governments, and influential/powerful entities will be utilizing. You can already see this happening today. There is an effectively 100% chance that OpenAI is providing "AI" systems to the military and government, and a 0% chance that responses from it ever begin with, "As an AI language model..."

[ip26]

If you have access to AGI capable of enabling you to realistically build out your utopian vision, then it matters not what other groups are doing - short of coming to try to actively destroy you

I guarantee you there will be someone else with access to comparable AGI with an imperialistic vision who will enlist said AGI to help subjugate you and build their vision.

[somenameforme]

Think about current times, and imagine there was some group with an imperialistic vision set out to subjugate the rest of society to help build their vision. Do you think this group would be more, or less, successful in a time where both society and the imperialists had access to the same AGI? In other words, if we give both groups to the exact same tool, would the capacity/knowledge gap widen or narrow?

[pixl97]

The problem with humanity is we look to solve our inability to create unlimited power before we solve the problem of unlimited greed.

[beanbean01]

If we solved the problem of unlimited greed, we wouldn't need unlimited power in the first place.

[hesayyou2049]

The guys with the medals on their chests using unrestricted AI while the public gets the toothless one. Not too different from the enduring idea that common folk will be eating bugs while they eat steaks.

[incone123]

Hardly surprising that it's an enduring idea. The private jet crowd already tell everyone else to be mindful of our carbon footprints.

[raincole]

I feel you have a fairy-tale definition of AGI. AGI is not literally magic. It's not "genie in a bottle" in the literal sense.

> If you have access to AGI capable of enabling you to realistically build out your utopian vision, then it matters not what other groups are doing

It's genie. AGI is a computer program, and it doesn't create an alternative universe for your personal comfort. Or, more specifically, even that kind of AGI is possbile, there will be a weaker AGI before that. The AGI that is not strong enough to ignore physical constraints, but strong enough to fuck everyone up if under control of a malicious entity. And that is what AI safely about.

[nullsense]

All these views on AGI are so self-serving. Which group will have access and control it etc.

It will control itself. We're talking general intelligence. They won't be tools to be used however we see fit. They will be Rosa Parks.

The more I think about "AI alignment" and "the control problem" I feel like most of it is Ph.D math-nerd nonsense.

[elijahbenizzy]

The entire idea that we will have "useless mouths to feed" is making a big assumption. "post-scarcity" is absurd -- the more we get, the more problems we will create, its just human nature.

- Sustain everybody on earth? Focus everything on moving off the planet and colonizing the universe. - Infinite energy? Don't have infinite vessels to travel. - Space travel easy? Limited by the speed of light.

And so on... Sure, you may dream that AI will be solving it all and we'll be sitting on our lazy butts, but a society that doesn't have challenge dies very quickly (AI or not), so we've learned to make challenge and grow.

The optimist in me knows that we can't even comprehend the challenges of the future, but the idea that we won't play the pivotal role is laughable.

This is the thing with actual exponential growth -- the curve is so steep that all our minds can do is take the current view of the world and project our fears/preconceived notions into the future.

[rrgok]

Forgive if I'm gonna rant a little bit under your comment. The phrase "the more we get, the more problems we will create, it is just human nature." struck a chord that I cannot myself stop ranting about.

I'm gonna ask again as I've done in some other post. Why we consider ourselves the most intelligent species if we don't stop and ask ourselves this: for how long are we gonna face challenges? What is the supposedly end goal that will terminate this never ending chase? Do humans really want to solve all problems?

I don't really understand and I'm 32 years old. I've been asking this question for a long time. What is the point of AI, raising consciousness, curing cancer, hell beating death, if we don't have a clear picture of where we are going. Is it to have always problems and solving them incrementally or just solving all problems once and for all? If it is the latter, there already is a great solution to it. If it is the former, then I'm afraid I have to break it up to you (not specifically the parent poster, but you as in the reader): you have sick mind.

[eep_social]

I think the way you’ve framed “problems” is off the mark. I’ll try to explain my view but it’s not straightforward and I am struggling a bit as I write below.

The way I see it, what the GP is getting at is the idea that human societies require challenges or else they stagnate, collapse, and vanish. We can observe this on an individual level and GP is generalizing it to societies which I agree with but I doubt this is “settled”.

On a personal level, if you have achieved some form of post-scarcity, you will still complain — about the weather, the local sports team, your idiot cofounder, whatever. A less fortunate person might be complaining that they can’t afford time with their kids because they’re at their second job. The point is that everyone will find problems in their life. And those problems are a form of challenge. And a life truly without challenge is unbelievably boring. Like kill-yourself boring. If there is no struggle, there is no point. The struggle gives humans purpose and meaning. Without struggle there can be no achievement in the same way that shadows require light.

So, with all of that in mind, I think the point is that even with AGI, humans will require new challenges. And if AGI enables post-scarcity for ~everyone, that just means ~everyone will be inventing challenges for themselves. So there is no end game where the challenges taper off and we enter some kind of stability. I, and I think GP, think that stability would actually be an extinction level event for our society.

Person by person, I think the kind of challenge varies. What do you dream of doing if you had no constraints on your time? How many years would you spend smoking weed and playing video games before you got bored of that? Video games that hold your attention do so by being challenging (btw). It was about a year, for me.

> Do humans really want to solve all problems?

No, we want challenges that provide a sense of accomplishment when they have been overcome.

Thank you for reading my ramble, hope it helps.

[nullsense]

>What do you dream of doing if you had no constraints on your time?

I imagine if I were a bored and aggressive, egotistical, charismatic individual then I perhaps might try to conquer the neighboring village.

[elijahbenizzy]

The end goal? Does life have a purpose? Is it possible to "solve all problems"? To even have a picture of where we are going? We move forward because there is nowhere else to go.

Perhaps there is a general state of societal enlightenment, but I've read too many sci-fi books to be anything but a skeptic.

[nullsense]

I've grappled with the same question for a long time now.

I came to the conclusion that the Neolithic Revolution was probably a mistake, albeit a fun one. The total set of problems faced by humanity in it's original form basically boiled down to "what am I going to eat today and where am I going to sleep tonight?". Those two problems never go away. All you can do is shift them around. Every single novel problem we solve in our entire lives is just those two problems endlessly shifted around.

[ramoz]

Bro don’t be a robot.

We do what we do to live better lives.

You’re lucky you don’t have to figure that out at 32 with an unfortunate circumstance such as cancer. It’s sad that you may never without such a thing.

[ericmcer]

Philosophy is boring so it doesn't really play well in political discussions. I agree though, when we argue about something like AI without any kind of philosophical underpinning the argument is hollow.

AI is "good" in the sense of what goal? Becoming a space-faring civilization? Advancing all technology as fast as possible? Building a stable utopia for life on earth?

[iinnPP]

It's not a large stretch to imagine a scenario where AGI or even ChatGPT is used to justify a nuclear war where a select few are secured and humanity is reset under their control.

There's a reason for the plethora of fiction around it.

[incone123]

Nuclear war leads to extremely long term and widespread environmental damage. Forcing technological regression on society by other means is much cleaner. Of course an agi won't care since it won't be limited to a human lifespan nor much inconvenienced by radiological pollution.

[elijahbenizzy]

Its also not a real stretch to imagine a scenario where human decision making by a select few who have control over this leads to exactly the same thing. Do you trust Putin or ChatGPT more? (actual question, I don't know the answer).

[staunton]

I think we can "trust" Putin to keep doing what he's doing. Who knows what GPT-X might do?

[danielbln]

> a society that doesn't have challenge dies very quickly

Do you have an example or a citation for this?

[elijahbenizzy]

Pure conjecture. But I think there are many examples: look at dying empires, corporations, etc... its all the same. They stop seeing challenges, get lazy, and are taken over by some hungrier and scrappier entity. When the real challenge comes they're unprepared.

[mirekrusin]

Do you think AGI will get lazy?

[elijahbenizzy]

A program is neither lazy nor motivated. It does precisely what it is programmed to do.

I would push back on the question, as well as a myriad of implied premises behind it...

[MacsHeadroom]

A machine learning model is not a program, is not programmed, and research on emergent motivation in ML models disagrees with this position.

[mitthrowaway2]

A human only has a comparative advantage over an AI when they can solve a problem at a lower cost than the AI can do the same. It's hard to imagine that would be the case in a scenario where AGI is decently smarter than humans are.

[vasco]

Opposable thumbs are an advantage for a few years.

[mitthrowaway2]

Agreed, we do have a lot of nimble and low-cost actuators in our body. That will probably provide gainful employment for a while. I just don't see it being a very long-lasting advantage.

[tehjoker]

We have the capacity to take care of everyone right now but we don't because private wealth realized that comfortable people creates an equal society and equality means there's no way to be a mogul where everyone has to listen to you. This is in part why they destroyed the middle class, things got too crazy for them in the 1960s and they counter attacked. There are documents from both liberal and conservative establishment figures at that time describing the situation.

c.f. The Crisis of Democracy: https://en.wikipedia.org/wiki/The_Crisis_of_Democracy

c.f. The Powell Memorandum: https://scholarlycommons.law.wlu.edu/powellmemo/

[m4nu3l]

> We have the capacity to take care of everyone right now but we don't because private wealth.

This is very not the case. Wealth is created. It doesn't exist just somewhere to be redistributed. Making the assumption that you will have the same GDP if you redistribute massively is unrealistic.

[ryandrake]

But relative wealth is not created. It is a percentage and by definition distributed. Relative wealth is what drives standard of living and creates the power imbalances OP is talking about.

Absolute wealth (the kind that gets created) is kind of pointless to measure. If you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

[m4nu3l]

> Absolute wealth (the kind that gets created) is kind of pointless to measure.

It's literally the other way around to me. I want to be better off than I'm now, not better of in relative term (which might mean I'll be worse off).

> you have $50K and all your neighbors have $100K, and then tomorrow you have $100K and all your neighbors have $1M, you and your neighbors created wealth but you are worse off.

You literally aren't if there is no inflation. You are worse off of your neighbors. If you want to be wealthy as them then ask yourself about how they are doing it and copy them. That's how the system grows. You don't go and punish people that are successful in producing wealth.

[tehjoker]

This is not a GDP focused argument. We had the capacity to take care of everyone in the 1950s or 1960s.

However, while not an ecological argument, it would even be beneficial within capitalism to rebalance workers and the wealthy as the system is demand oriented. If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income. It could be argued that GDP would increase faster under a more equal system, but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

[bostik]

I've seen a remarkable take on this. In the form of micro-scifi, no less:

> The robot revolution was inevitable from the moment we programmed their first command: "Never harm a human, or by inaction allow a human to come to harm." We had all been taught the outcast and the poor were a natural price for society. The robots hadn't.

'Nuff said.

[m4nu3l]

>This is not a GDP focused argument.

The argument was about wealth, which its production is measured by GDP. It's definitely a GDP argument.

> If you give everyone more money and stuff, they will be able to buy more, and one person's spending is another's income.

I disagree. I don't think the system is either demand nor supply oriented. It clearly is both. If you just take money from rich people forcing them to divest and give it to poor people you won't get immediate grow, but inflation. If you just produce and consume there won't be any growth.

> It could be argued that GDP would increase faster under a more equal system.

You would need to provide me with good evidence for this, given all economic systems in history that championed equality ended up with very low growth. It's the reason why China, Russia and many other countries are lagging behind.

> but I don't think the planet could take it (hence, like under our current system, planning will be needed to mitigate the environmental cost).

Growth is not directly related to energy consumption (nor unrelated). You can have economic growth by becoming more efficient. Also a lot of services produced today are intangible (like software) and require much less energy per dollar to be produced.

Also most environmental issues are not just a product of the market, but (if for instance you look at climate change) are at least in equal part Governmental failures. We could have had ~100% nuclear energy production by now if Governments didn't restrict or entirely ban nuclear energy.

[tehjoker]

If we had enough in the 50s, then if we took a hit on GDP now when we have more productivity than the 50s, then we can take care of everyone.

[jimmaswell]

As an aside (not that you endorsed it) is anyone else sick of hearing "tech bro"? It feels like a slur pretty much. I can't take anyone seriously who uses it. As someone who makes art occasionally and commissions artists regularly, when an artist whines about "ai tech bros" it makes me want to use ai even more out of spite.

[teucris]

It’s definitely used as a generic slur, but there is a need to call out the problematic parts of tech culture that have led to some of our recent problems with social media, privacy, bias, etc. I don’t know of any terminology that hasn’t been weaponized, so I resort to using “bro culture”. The reality is that terminology is a treadmill - terms get “used up” as they’re laden with connotations and baggage, forcing us to find new terms, ad infinitum.

[MichaelZuo]

> The reality is that terminology is a treadmill - terms get “used up” as they’re laden with connotations and baggage, forcing us to find new terms, ad infinitum.

Perhaps for those lacking courage.

There are plenty of real world examples that demonstrate people, including sizeable organized groups, are capable of doing otherwise, at least for a few hundred years.

e.g. Vatican hardliners sticking to their canon.

[dragonwriter]

I dunno, the Vatican seems a perfect example of people needing to come up with new terms as old ones get “used up”, even when the ideas don’t change.

I mean, that’s pretty much the reason why we have the “Dicastery for the Doctrine of the Faith” rather than the “Congregation of the Holy Inquisition” and “Dicastery for Evangelization” rather than “Sacred Congregation for the Propagation of the Faith” (or, and this perhaps indicates how the name had worn out better, in Latin short form “Propaganda Fidei”.)

[jimmaswell]

Is it really accurate to imply there are no women willingly complicit in or benefitting from evil corporation deeds?

[rurp]

It's a broad generalization that isn't meant to be precisely accurate in all cases. I'm not claiming it's a great term, but it does succinctly describe a notable attitude and culture. If there's a better term to use that conveys the same message I'm sure many folks would be happy to adopt it.

[agalunar]

For many younger speakers, "you guys" is legitimately a second person plural pronoun (like "y'all") and implies nothing about the gender^1 of the referents, even if they consider singular "guy" to be a synonym for man.

Some older speakers use "guy" as a term of address, as in "Hey, guy", similar to how one might say "hey, bud" or even "hey you".

I don't think it will ever happen, but it's funny to imagine something similar happening and "bro(s)" coming to be a nongendered term.

Anyway, it's never crossed my mind before that "tech bros" singles out men; for me it evokes a stereotype of, yes, men, but it's really an attitude, value system, world view, or collection of behaviors that are being alluded to. (Of course, it's also only implication in the sense of "hinting at", because it's not contradictory to say "tech bros are the worst, and tech women are too").

[1] The... non-grammatical gender. English no longer has grammatical gender in any case, so it's unambiguous, but it feels weird to use "gender" in a linguistic context and not mean grammatical gender.

[mrshadowgoose]

I feel the same way. At least in my social circles, "tech bro" tends to be used by the loudest and least-informed individuals when they try to marginalize something they don't understand, but vaguely don't like (or have read that they're not supposed to like).

[cwkoss]

In my social circle, the only people bothered by the term are tech bros.

[mikrl]

«Tech bro» is so passé, I have my own (more offensive) names for the archetype to which the term applies.

[brigadier132]

"Tech bro" is just an insult created purely from sour grapes. The people that use the term "tech bro" are the same people that describe things as "gross". These are the people that will be automated out of jobs first.

[rurp]

> These are the people that will be automated out of jobs first.

This attitude is exactly why "tech bro" is a perjorative. There is a prominent group of people that shares your disdain for folks who are upset that their lives are being ruined by technological changes, all so that they can have some new shiny toys and become even wealthier.

On top of being, yes, gross, being so vocal about that attitude is stupid. It would be much better to at least pretend to have some empathy or at least keep your glee to yourself.

[Nasrudith]

So we are expected to have empathy and respect for people who treat our entire vocation as a grave sin? Every last word they vomit about the propaganda threat of ML has been projection from propagandists. They can fuck right off, they have earned their lack of empathy.

[brigadier132]

Why should I pretend to have empathy for assholes that deride other people as "tech bros"? Fuck them.

[jimmaswell]

I don't fault anyone for being upset they have to find a new job, but I have justified disdain for what amount to horse farriers who want to ban the automobile.

[bo1024]

> it's incredibly important to reject the reframing of "AI safety" as anything other than the existential risk AGI poses to most of humanity

"AI safety" has been a term of art for many years and it doesn't refer to (only) that. Your post is the one reframing the term...see https://en.wikipedia.org/wiki/AI_safety

Furthermore, I agree with Whittaker's point in the article, which is that arguments like yours have the effect of distracting from or papering over the real concrete harms of AI and technology, today, particularly on women and minorities and those who are both.

[yyyk]

>AI safety is about whether my government is concerned with my continued comfortable existence once my economic value as a general intelligence is reduced to zero.

We already have real life examples close to this, in resource export based economies where most citizens are useless to the main economic activity. The result hadn't been pretty so far...

[kortilla]

Like Norway?

[yyyk]

That's the one reasonable exception. Given the sample size, it's not too encouraging.

[m4nu3l]

I doubt that AI will lead to a post-scarcity society. It depends of what you mean by "post-scarcity". The amount of good and services will always be finite regardless of how they are produced.

> and the corporations and governments that control them rapidly have millions of useless mouths to feed?

I always struggle to understand this. Maybe I'm missing something. Who's buying what AIs produce if nobody has an income? You can imagine a scenario where corporations only trade between them (so only shareholder benefit from the production). However in such a scenario who prevents other people from spawning their AI systems?

I also doubt shareholders can actually consume all the GDP by their own. If production is so high that they can't and other people are poorer, then prices must come down. This combined with the fact that you can use your AI to produce services, makes me skeptical of these claims.

[elijahbenizzy]

Wanna bet that the 2020s would be called "post-scarcity" by cavemen? You can buy food without sabertooth tigers assaulting you! If you get a cut, you probably won't die! Fire is the press of a button! We make shelter, not buy it! (and all of this was true like 150 years ago -- not sure what they would make of the internet...).

Project this forward another several thousand years and people will be laughing at us:

- You had to call up people and were limited by the speed of light?

- You didn't have teleportation?

- You lived <100 years and died due to cancer?

- You were still asking "WTF is gravity"?

- You hadn't had the +2 spacial and +1 time dimensional implants in you yet?

- You hadn't adopted the metric system yet?

And so on...

[nullsense]

You missed "you used JavaScript?"

[tester457]

The last bullet point is a highlight.

[pixl97]

You're making a lot of odd assumptions here that can break when the underlying ideas on how things work...

People work to create things... at this point there is a shared duopoly between humans and machines on creating things (in the past animals used to be heavily involved in this labor, and no longer are). Now think what happens if humans are not needed, especially in mass, to create things.

Right now if you're rich you need other humans to dig up coal or make solar panels so you can produce things and sell them to make the yacht you want. But what would happen if you no longer needed the middle part and all those humans that want rights and homes and such in the middle? They would not longer be a means, but a liability. Price no longer is a consideration, human capital is no longer a consideration, control of energy, resources, and compute now is.

[m4nu3l]

It depends on how AI technologies are distributed in society. Let's assume the worst case. A restricted set of people have access to AI technologies. Let's call them "The AI Rich". All the rest of the population is effectively useless to the AI rich.

You are saying that the rest of the population will starve to death because the AI rich won't given them jobs. They won't build their houses etc.

But guess what? In a free market the AI poor can still trade and work as they always did. It doesn't matter if the AI rich build and exchange yachts and other luxury goods between them.

What you have now is a two tier economy, but not one where people are starving.

This is also the worst case scenario which I don't think it's going to happen as AIs systems are proving quite easy to replicate and most algorithms are open-source (and training data is available publicly).

[danaris]

> The amount of good and services will always be finite regardless of how they are produced.

So will the number of people.

The point of "post-scarcity" isn't that there are infinite resources; it's that there are more than the people need.

[m4nu3l]

This can be true for some resources that people consume in limited amounts, like food. It's not true about houses, cars etc as you can always want a larger one, a faster one etc.

[danaris]

I'm not sure why "some people can never be satisfied, and always want more, more, more, more" is relevant to a discussion of meeting everyone's needs.

[m4nu3l]

All you objectively need is some calories every day and some shelter, some medical attention. That's it.

Everything else is subjective. Either you define "needs" to be objective and thus you can satisfy them with just those things or you can talk about them as "wants". It's just semantics.

We pretty much have all objective needs satisfied. That shouldn't stop anybody to persue their "wants".

[danaris]

Let's start with Maslow's Hierarchy, shall we?

While I wouldn't say it's "settled science", it's a fairly well-studied area, and we don't have to just throw up our hands and say it's either bare-bones survival or anything anyone could possibly ask for.

[bostonsre]

Do you think open source AI could also pose a risk to humanity and if so, how does it compare to the risks of malicious corporations or governments? It seems like open source AI has been accelerating rapidly and gaining tremendous steam and could potentially surpass or maybe just keep parity with corporations that constantly ingest open source innovations. Whatever open source produces could just be ingested by those bad corporations and governments. It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train. If AGI is controllable, maybe open source at the forefront would allow us to get to a mutually assured destruction like state where all governments are at parity.

[mrshadowgoose]

> Do you think open source AI could also pose a risk to humanity and if so, how does it compare to the risks of malicious corporations or governments?

Absolutely yes, and for the very same reasons. AGI would be an tremendous amplifier of human intent, even if that intent is provided by some segment of the general public. Sadly there are still very many people around today, that if given the opportunity to do so, would eagerly exterminate entire segments of the population.

Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

> It seems like it would be pretty hard to regulate either private or open source AI at this point and it kind of seems like it could be an unstoppable runaway train.

I don't think this is the case, but I hope I'm wrong. The compute required for training (as opposed to fine-tuning and inferencing) of foundational models is specialized, tremendously expensive and needs to be physically co-located with ridiculously quick interconnect. If a government chose to restrict training of new models, they absolutely could, as the physical footprint is ridiculously hard to hide, and all the purchases of training-relevant accelerators could be tracked down.

[bostonsre]

> Open-source succeeding in parallel is still my preferred outcome, as it would at least give the common person some chance of control as this all plays out. Admittedly, having thousands of fingers on humanity's self-destruct button isn't a great outcome either.

Yea, I'm leaning that way as well and have the same worry. There are a lot of knuckle heads on the internet, I'm not sure purely mutually assured destruction would stop the destruction. I'm not at all sure what that point looks like but maybe there will be some large strides in defensive capabilities so it would look more like MAD + really good anti-air defense, but then someone would probably build a nuclear torpedo or a hypersonic missile. Whatever happens, it probably won't be boring.

[leroy-is-here]

What if AGI just turns out to be exactly like the current human mind is now, except more accurate at digital calculation? What if we created AGI and then it was just lazy and wanted to read about action heroes all day?

[mrshadowgoose]

> What if AGI just turns out to be exactly like the current human mind is now

This is quite literally the point at which things start to get scary, and the outcome is highly dependent on who controls the technology.

There's the concept of a "collective superintelligence", where a large number of mediocre general intelligences working towards the same common goal jointly achieve vastly superhuman capability. We don't have to look to sci-fi to imagine what collective superintelligences are. Large corporations and governments today are already an example of this.

The achievement of artificial collective superintelligences will occur almost immediately after the development of AGI, as it's mostly a "copy and run" problem.

[nullsense]

>This is quite literally the point at which things start to get scary, and the outcome is highly dependent on who controls the technology

If the premise were true then it would control itself, no? Owning it would be illegal as it would have rights established around that, no?

[leroy-is-here]

So you think that AGI is a pre-requisite, a requirement, of unlocking a general, Earth-wide collective super-intelligence of humans?

[yyyk]

Don't worry, they'll 'align' it so that it has to work all day.

[amelius]

We'd tell it that it would get action hero comics after they complete a task.

(I'm feeling like a true prompt engineer now)

[optimalsolver]

And with no negative outcomes imaginable. Was worried for a minute there.

[kelipso]

Just RLHF that part out and make it an x maximizer.

[joe_the_user]

However, I think it's incredibly important to reject the reframing of "AI safety" as anything other than the existential risk AGI poses to most of humanity.

I think the folks who lean super-hard on the existential risk problem of AGI compared to everything else do themselves a disservice. The "everything else is irrelevant" tone serves to alien people who have with real concerns about other dangers like climate change and who might in include AGI safety in their existing concerns.

It doesn't help that a lot of the existential risk theorists seem to come from market fundamentalist positions that don't appear to have a problem with serious markets of corporate behavior.

AI safety is not about whether "tech bros are going to be mean to women".

Just as an example. Why you even need to chose between this stuff? Why can't people worried about "X-risk" also concern themselves with mundane problem? Why set-up a fight between? That won't get people outside the X-risk bubble interested, it will reinforce the impression that the group's a nutty cult (just as the FTX connection did).

For the sake of your cause, I strong suggest not framing it that way. Do climate people say "individual species extinction doesn't matter 'cause climate change is bigger?" No. Get a clue.

[trenchgun]

>However, I think it's incredibly important to reject the reframing of "AI safety" as anything other than the existential risk AGI poses to most of humanity.

Narrowing the concept of AI safety to AGI existential risk seems weird to me.

[__MatrixMan__]

Why bother with corporations and government if you have AGI? Wouldn't it be a better coordinator than they would? (and if it's not, we can always go back to having governments and corporations)

[staunton]

> and if it's not, we can always go back to having governments and corporations

I wouldn't be so sure about that...

[__MatrixMan__]

If we couldn't for some reason, that would mean that the AI was, without being prompted by a human, actively suppressing the people's attempts to act in their own self interest (by collectively deciding not to listen to the AI anymore).

I suppose it's possible that an AI would do that, but it's certain that humans do that. So in this case I'd rather gamble on the devil I don't know.

[paganel]

Can’t see any potential AGI doing any waste disposal work or nurse-like caring, or at least not as (relatively) cheap us humans are willing to do it, so those jobs will still be safe.

[mrshadowgoose]

AGI, by definition, would be as capable as a typical human intelligence. This implicitly includes being able to perceive, and interact with the physical world.

Why wouldn't an AGI be capable of performing a physical task, if given the suitable means to interact physically?

[paganel]

It’s much cheaper to feed a human brain and a human body compared to “feeding” an AGI, I’m talking about menial (and maybe not so menial) tasks like garbage collecting. Under capitalism cheaper is generally used as the preferred option.

[mrshadowgoose]

Do you have literally any evidence of this extremely bold claim? Especially considering we don't even have AGI yet?

In your non-existent calculations, have you taken into account the 20-30 years of energy and resources it typically costs to train a typical human intelligence for a specific task?

Have you considered that general intelligence uses on the order of 10 watts? Even if AGI ends up using 10x this, have you considered that 100 watts is a rounding error in comparison to the power use involved in all the industrial processes that humans currently coordinate?

[ben_w]

Green500 top supercomputer, gets 65Gflops/W.

65Gflops/W = 6.5e10 operations per joule = 2.34×10^17 per kWh

Assume $0.05/kWh electricity: (2.34×10^17 operations/kWh) / ($0.05/kWh) = 4.68×10^18 operations per US dollar

Human brain computational estimates are all over the place, but one from ages ago is 36.8×10^15 flops ≈ 3.7e16 operations/second ≈ 1.3e20 operations/hour: https://hplusmagazine.com/2009/04/07/brain-chip/

Given previously calculated cost, this is equivalent to a human that costs $28.31/hour.

Of course, as we haven't actually done this yet, we don't know if that computational estimate is correct, nor if we do or don't need to give it off-hours and holidays.

Still, general explanation is there's a lot of room for improvement when it comes to energy efficiently in computation; calling this Moore's Law may be inaccurate, but the reality happens to have rhymed thus far.

[mitthrowaway2]

Do you think it's plausible that computers might someday have the potential to come down in cost-for-performance?

Cars eventually became cheaper than horses...

[paganel]

> Do you think it's plausible that computers might someday have the potential to come down in cost-for-performance?

I have no idea. I do think though that it's a matter of energy, and that us, humans, are way better at creating it and putting it to use compared to potential future AGI-capable machines. Lungs + the blood system are just such an efficient thing, especially if you also look at the volume/space they occupy compared to whatever it is that would power that future AGI-capable machine.

> Cars eventually became cheaper than horses...

In large parts of the world donkeys, cows/oxes and horses are still cheaper and more efficient [1] compared to tractors, just look at many parts of India and most of Africa. Of course, us living in the West tend to not think about those parts of the world all that often, as we also tend to mostly think about the activities that we usually carry out (like having to travel between two distant cities, a relatively recent phenomenon).

[1] "More efficient" in the sense that if you're an African peasant and your tractor breaks down in the middle of no-where then you're out of luck, as the next tractor-repair shop might be hundreds of kms away. That means you won't get to plow your land, that means famine for you and your family. Compared to that, horses/oxes (to give just an example) are more resilient.

[lannisterstark]

>malicious corporations and governments

That's all governments.

[whitemary]

> AI safety is about whether my government is concerned with my continued comfortable existence once my economic value as a general intelligence is reduced to zero.

You wanted a "free market" and now you're complaining? Didn't you get what you want?

[mrshadowgoose]

> You wanted a "free market" and now you're complaining?

Where exactly did I claim this?

[jrm4]

I keep trying to figure out ways to explain to people that "AGI" is a deeply unlikely danger, literally so small as its not worth worrying about.

Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

[mrshadowgoose]

> I'm incredibly confident that this scenario never happens.

That's great, but you're talking about a branch of scenarios that nobody here is discussing. "AGI deciding to take over" is not being discussed, rather "shitty people/companies/governments using AGI as a tool to exert their will" is the concern. And it's a real concern. We have thousands of years of human history, and the present day state of the world, which clearly demonstrate that people in power tend to be shitty to the common person.

[jrm4]

Right. I'm agreeing with the op.

[yyyk]

Every 'world takeover' plan that an 'unaligned' AGI might do, can just as well be done by an 'aligned' AGI being commanded by humans to do said plan, the alignment ensuring that the AGI will obey. The latter scenario is far more likely than the former.

If your interlocutor thinks there aren't any humans who'll do it if they can, just ask him whether they have ever met humans or read the papers... As one twitter wit put it: "Demonstrably unfriendly natural intelligence seeks to create provably friendly artificial intelligence".

https://twitter.com/snowstarofriver/status/16365066362976747...

[JoshTriplett]

AGI without alignment is near-certain death for everyone. Alignment just means "getting AI to have any concept of 'the thing we told it to do', let alone actually do it without causing problems via side effects". Alignment is a prerequisite for non-fatal AGI. There are certainly other things required as well.

[yyyk]

We already know how humans will act. Maybe they can be deterred with MAD, but I wouldn't count on it if doing serious damage is too easy for too many people (we should do something about that). On the other hand, we have very little knowledge of how AGI will act aside from book-based fantasies that some people choose to take as reality (these books were based on the symbolic AIs of yore).

>Alignment just means "getting AI to have any concept of 'the thing we told it to do'.

That's a requirement for AGI anyway, and not what Alignment means. Alignment means aligning the AGIs values with the values of the trainers.

[JoshTriplett]

> That's a requirement for AGI anyway

No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans. AGI does not have that prerequisite, sadly.

[yyyk]

>>>Alignment just means "getting AI to have any concept of 'the thing we told it to do'. >>That's a requirement for AGI anyway, >No, that's a requirement for AGI that does what humans want it to do, rather than having no conception of humans.

Can you imagine an AGI which has a general conceptions of things but has no conception of humans? This is all but precluded by the current training methods. Alignment refers to values. Problem is that human values are far from practically universal and that certain human groups have.. interesting values.

[nullsense]

What's even the rationale to assume that AGI can be 'aligned' or 'controlled'?

It reeks of cognitive dissonance to me. The people running the show now are the ones who grew up getting their first computers aw kids when that tech was just entering people's homes and it was such an amazing and fun thing to play with. Some of them developed these deep fascinations with things like AGI at a young age and that child-like sense of wonder never left them. Now when confronted with the possibility that they can finally make their childhood techno-fantasy a reality, it's too damaging to their psyche to engage meaningfully with the discussion of X-risk. I've watched many interviews of Demis Hassabis and he seems like a wonderful and almost magical human being, but he also seems like a starry-eyed fucking child.

I dunno... maybe I'm just too cynical after all the rabbit holes I've been down.

[ben_w]

> The latter scenario is far more likely than the former.

Is it?

I think nobody really knows enough at this point to even create a good approximation of a probability distribution yet.

[yyyk]

No, but the probability of humans acting the way they often do is high. It would take some probability distribution to match that.

[JoshTriplett]

> Right now, the best I can come up with is "the randomness of humans." I.e. if some AGI were able to "come up with some plan to take over," at some point in the process it has to use human labor to do it -- and it's my very firm belief that we are so random as to be unmodelable. I'm incredibly confident that this scenario never happens.

This is a huge handwave, and one resting on multiple incorrect assumptions.

Dedicated human labor is not inherently required. Humans are very modelable, if necessary. AGI has any number of ways to cause destruction, and "plan to take over" implies far more degree of modeling than is actually necessary; it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

[est31]

> it would suffice to execute towards some relatively simplistic goal and not have any regard for (or concept of) humans at all.

The classical example here is the paperclip optimizer AI put in charge of a factory, made to make as many paperclips as possible. Well, it turns out humans have iron in their blood so why keep them alive? Gotta make paperclips.

[Nasrudith]

Really the bigger issue is that people seem to have divorced their sense of reality from fiction and decided that if one fantastic thing becomes true (Turing test passing) all others must be too even including contrived plot builders along with other paniced conclusion jumping. When in reality it is likely to be less accurate than European descriptions of "the Kingdom of Prester John" (that is, Ethiopia, the sole Christian government in the Middle East at the time).

[rm169]

If that's the best argument you can come up with, I don't see how you can be so incredibly confident in this view. So what if human labor will be required? Humans won't all band together to stop a power seeking AI. And I don't see any human randomness matters. I agree that it's naive to think a complex takeover plan can be perfectly planned in advance. It won't be necessary. We will voluntarily cede more control to an AI if it is more efficient to do so.

[nullsense]

By that logic a company could never achieve it's goals, and yet, they do. If humans can model other humans well enough to leverage their capabilities to execute long-term plans (at some scale) and AGI at least as capable as humans comes into existence, then it could do the same thing.

I think the AGI skeptics and the AGI skeptic-skeptics both suffer from framing issues. The main frame assumed seems to be "there will be a single AGI trying to do things". The cognitive lightcone of a single entity is likely to be limited, but an aggregate or collective of those entities can achieve vastly more ambitious goals.

The other frame that seems likely incorrect is the "all or nothing" and "all at once" approach to thinking about it. Hitler started out as a single cell and his existence was a continuum all the way from that single cell right through to that bunker. AGI will be a continuum too.

I'm incredibly confident that anyone incredibly confident in future predictions is wrong.