[noelwelsh]

The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".

If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.

TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"

There's also a need to consider the rights that this new intelligence should have.

[f6v]

> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world

What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?

[Marha01]

Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.

Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.

[LunaSea]

> which is why obesity is a much bigger issue than hunger today

Obesity is much more related to the type of food than the quantity.

Many developing countries have obesity issues due to scarcity of fresh and healthy food.

In some places coca cola is cheaper and / or more available than drinking water.

[asdff]

Hard to imagine that a 24 rack of coca cola is actually cheaper than an iodine tablet or a couple drops of household bleach or you know, boiling water.

[rubyn00bie]

It can be when you don’t have access to boiling water, bleach, iodine tablets, and/or the water you have access to is extremely contaminated with chemicals because of pollution.

[asdff]

If you can find burnable material you can boil water

[squidbeak]

> Hunger is present pretty much only in conflict zones.

How do you explain the existence of food banks in peaceful first world countries?

[StilesCrisis]

Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.

[felixgallo]

Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.

https://www.whitehouse.gov/wp-content/uploads/2025/05/Cuts-t...

Civilization can't rely effectively on systems that are this fragile.

[StilesCrisis]

> Civilization can't rely effectively on systems that are this fragile.

That's American politics in a nutshell. We've spent 250 years assuming scruples and common decency would be sufficient.

[OttoVonBizark]

The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank

[xyzzy123]

Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.

At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.

The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.

Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".

It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.

[cornholio]

It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.

So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.

[fragmede]

https://www.marketplace.org/story/2025/11/26/how-a-market-sy...

There's a fascinating points system food banks use on the back end to allocate which food goes to which bank.

[cassianoleal]

> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.

Is that a fact? Do you have references to back it?

[jandrewrogers]

The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.

You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.

[pixl97]

I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).

And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.

[ratelimitsteve]

As part of the solution to the hard problem of food: distribution.

[hgomersall]

There's plenty of food, just not the political will to make sure everyone can afford to buy it.

[overgard]

Housing scarcity is highly artificial because people are treating houses like speculative assets. And the NIMBYs don't help

[cornholio]

A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.

> the limited availability of land zoned for housing.

A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.

[bloppe]

It's a limited area of desirable land zoned for housing. You could build all the tiny homes in the world in bum frick nowhere, but the unhoused would rather be homeless in a big city for various reasons. I probably would too.

[cornholio]

This is another reframing of the problem. What makes the area desirable is previous investment society has made there, to the benefit of some and the exclusion of others; the scarcity is created by political choices, not intrinsic.

[dinkelberg]

> obesity is a much bigger issue than hunger today

Hunger is acute suffering, mostly by people who cannot change anything about it, while obesity is more of an epidemiological problem and can in principle be avoided by eating (or in the case of dependent persons, feeding) less.

[turtlesdown11]

> Hunger is present pretty much only in conflict zones

https://www.fao.org/interactive/hunger-map/en/

[seltzered_]

I recommend doing some reading about the inputs modern agriculture depends on (e.g. chemical fertilizer) then thinking about the issues they're associated with beyond an anthropocentic frame (e.g. ocean dead zones).

[allears]

Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.

[Marha01]

Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.

[watwut]

> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.

It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.

The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.

[overgard]

Uh, I think threatening the work of a large swath of humanity would make a lot of people poor? The rich historically have never shared the wealth until they're forced to.

Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."

This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.

[bloppe]

I support progressive income tax and very high capital gains tax. I do not support wealth tax. There are very sound economic and technical arguments against wealth taxes. You can oppose the wealth tax proposal while supporting higher taxes for the rich.

Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.

[gmuslera]

We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).

Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?

The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.

[SirHackalot]

> There's also a need to consider the rights that this new intelligence should have

A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.

[tavavex]

Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?

[bloppe]

Very agree.

I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.

People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.

But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.

That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.

[pixl97]

>convincingly conscious

The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.

>A survival instinct is irrelevant when you're backed up to Google Drive

And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.

[pixl97]

>without the need for these reward and punishment mechanisms?

AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.

So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.

These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.

>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?

Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!

[energy123]

Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.

[DFHippie]

Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.

[skirmish]

Quite right, there is nothing to discuss since qualia don't exist. See Daniel Dennett.

[SirHackalot]

It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open. If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…

[antonvs]

> To me, AGI implies consciousness, it implies qualia.

I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs.

Why should a more general kind of intelligence imply consciousness?

[SirHackalot]

I’m saying imagine something like Data from Star Trek, something that defies human explanation. Does he have quaila? The humans who work with him everyday seem to think so. LLMs currently are humonculi, if even that. They’re not even that. I’m definitely not saying your LLM is your slave.

[pixl97]

>AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions

Then it's pretty useless for rational discussion.

Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.

Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.

> make a comeback to justify AI slavery for hundreds of years

I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.

[nradov]

It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.

[SirHackalot]

You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.

[nradov]

I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.

Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.

[jay_kyburz]

Humans are just walking Electrons.

I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.

We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.

[SirHackalot]

Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...

You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.

[Diogenesian]

I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.

I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.

[miyoji]

> They don't have free will, and I don't mean in any especially philosophical / theological sense

Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.

[SirHackalot]

An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.

[Diogenesian]

I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.

My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.

[pixl97]

>none of us will live to see a computer that's meaningfully smarter than a goldfish.

I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.

[andy_ppp]

The A(G)I can tell us if and how it needs to be regulated :-/

[mikeyouse]

My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.

What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?

https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"

[Marha01]

> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.

The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.

At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.

[hackinthebochs]

As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.

[ralfd]

Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.

Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.

Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.

[mikeyouse]

> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.

From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.

As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]

> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.

Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?

[1] - https://taxpolicycenter.org/sites/default/files/publication/...

> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax liability by 14.5 percentage points."

[whimsicalism]

The fundamental issue is that if we really get something like this, scarcity will still exist. There will still be scarce things people want.

But the motivating justificatory structure for any inequality in allocation will have completely evaporated.

[throw4847285]

Am I crazy or does this just real like secular eschatology? What evidence do you have of any of this?

[altcognito]

Particular categories of land will always be scarce by definition. Not all of us can live in off the coast of the Mediterranean in a sprawling villa.

[noelwelsh]

There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.

There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.

[overfeed]

Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.

[ponector]

Both AGI and nuclear fusion are a few short years away.

[antonvs]

For commercially viable fusion energy production, you can make a strong case that it's not possible in principle.

It's more difficult to make that case for AGI.

[danans]

> There's also a need to consider the rights that this new intelligence should have.

Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.

[antonvs]

Why?

[javierluraschi]

Model cards won't save us: https://x.com/javierluraschi/status/2077147606791430372?s=20