[godelski]

> It's more that nukes

I'm sure that this is part of "long peace" but we can't discount globalism. At the end of the day a lot of war is about economics. When countries become highly dependent upon one another, including enemies, it becomes much harder to actually go to war with one another. And as you point out with the nukes argument, that cost for going to war has also increased. So it's often far easier to war via economic means rather than physical.

There's also an (much more debatable) argument to be made that the so called "world police" does not have neighboring land that it covets. America doesn't have much need or want to grab land from Canada or Mexico, and doing so wouldn't have huge economic impacts on it. But such a situation is by no means true for Europe (and arguably Russia or China). I mean this is why Europe was fighting for the last... well however long humans have been in Europe (same being true for east Asia and really most of the Eurasian continent).

[p1esk]

Off topic, but I would like to hear your opinion on the impact of the AI progress on jobs, globally. Let’s assume, for simplicity, that GPT5 will actually be significantly better (eg similar improvement as what we had with 3.5 —> 4). And another assumption - it will be possible to put GPT5 level model into a humanoid robot, and train it to do a variety of basic physical tasks.

If we assume all that, and I realize it might not happen any time soon, same like with self driving cars progress, but *if* there’s strong and quick progress, what will happen to job market, unemployment, economy, and the society as a whole?

[godelski]

1/2

Sure, this is probably nearer to my expertise. I research ML (generative models) and my partner is an economist, so we have these discussions quite a lot. I'll try to keep it short. The tldr is I don't know and I'm pretty sure no one knows, and I'm even more concerned about the lack of discussion.

First off, I'd highly recommend watching the recent Dwarkesh video with Francois Chollete[0]. I normally wouldn't suggest Dwarkesh, but Francois is an oddball for that podcast. The reason I suggest this is to understand the difference between AGI and ML. It's probably important for the framing and making accurate predictions here. So while I don't think AGI is on the horizon (it could be, but we're reinforcing the railroad rather than exploring other paths), I do think there is still quite a potential for huge economic disruptions and entire paradigm shifts. You don't need AGI to get significantly closer (maybe even all the way) to post scarcity.

For about a decade now I've been asking a simple question and I encourage others to ask it of people that they know. I need no credit, I need people thinking about this question, and to be quite serious about it.

  How do you function as a society if 10% of the workforce is unemployable?

There's a wide variety of ways this can be framed and I encourage you to explore those. 10% is arbitrary, but chosen because both 1) people think of that number as small and 2) that number represents depression level of unemployment rates (this balance seems to be optimal for initial conversations, so choose an appropriate number for who you talk to). But the reason I started asking this question is I was wondering "how do you transition to post scarcity?" Because in that framework, those jobs are not coming back. But that doesn't necessitate that there are no "jobs," but they wouldn't be in the conventional sense (see Star Trek for one version of this).

I think post scarcity is obtainable and it is the number one problem humans should be working on right now. It comes with unimaginable benefits, but it also comes at potentially huge costs if we don't implement it correctly. You are right to be worried. But I think the difficulty here and what I often face when trying to ask this question is that it looks simple. UBI is by far the most common answer, but the way people answer "UBI" is no different than "wave a magic wand." There are many ways to implement UBI, many ways to distribute capital (which isn't only money. Remember money is a proxy, a fungible token). So the issue here is that often people will answer while only considering the question at a very surface level and then be satisfied and move on. This is a grave mistake, and we need to dig down into the details. There are many rabbit holes to dig into within this question, and I encourage you to go down some, but will also say that the question is so complex that I am quite certain that no single (or even small group) of human(s) could sufficiently resolve it.

[0] https://www.youtube.com/watch?v=UakqL6Pj9xo

[godelski]

2/2

These things not only include such aspects of distribution (if you choose to keep people alive (I've never heard anyone answer "let them die" fwiw. Even in the most libertarian groups)), which includes not only food, but even things like housing. You have to consider the immigration/emigration with whichever locality accomplishes this feat first. Peoples' psychology and how they will find meaning in their lives. Aspects like that it is quite possible that you can eventually have the inverse of the question with "how do you formulate a society when you require 10% of people to work, and no more?" (especially considering that many of those jobs would likely be undesirable). I'll let you ponder others.

So in this decade that I've been asking this question (often also talking about the motivation of transition to post scarcity), I've yet to receive a satisfactory answer. Everyone I've met, from many backgrounds and a wide range of intelligence levels (with many FAR more intelligent than myself) stumble upon answering this once any bit of complexity/nuance is brought up (see UBI above). Best I've heard is creation of jobs programs and a large increase of entertainers (but seems ML is coming after that too. But then again, we like to watch humans play chess against one another far more than machines). None of us were even satisfied with those answers more than "these might be enough to starve off the worst aspects of the transition, but certainly not enough."

It is extremely important to remember that it will not only be low skilled labor that will be displaced or automated away, but there is quite a bit of high skill (which is why retraining can be an unsatisfactory answer, because you're potentially pushing someone from a $300k+/yr job to $50k/yr and asking them to spend a few years to reskill themselves to do even that). In fact, because these machines aren't generalist agents, it actually makes high skill jobs the more likely ones to be displaced (specifically highly technical skills that are fairly routine). I find it common for people to think it will just be the "people already at the bottom", and I think this is often because people feel that they aren't and are mentally protecting themselves.

But I want to answer my personal belief on one part: what will people do with their lives. Because it is also the motivation to pursue this path.

I for one think post scarcity can create a new renaissance. That people can be the most human they have ever been. I think maybe at first many will slack off and just "veg out." That they'll party, travel, and do other such luxury things that we may not consider productive (but who cares? It is their life, right? The goal is to free people, and that includes freedom of the burden of labor). But I've seen very few people that actually don't get antsy after doing this for a few weeks. Essentially, they recover from their exhaustion and then feel like the need to do something, even if they don't know what that is (the exception to this, in my experience at least, is people who have or develop psychomotor retardation based depression. Which if our society calls these displaced people worthless, I expect this to be a fairly common outcome). But I think many people here on HN will realize that they themselves would not end up being static. The amount of Open Source software that exists by people who do this in their free time while already having a job that likely drains them, is too high. I expect more software to arise. I expect more art, more music. I hope we can form more communities, but we are ever increasing insular (but with more free time, maybe we'll go out more and talk to our neighbors more). I think it is hard to even know because such a society would look so different than ours and it is hard to not leverage our current frame of reference. Maybe it will be Wall-e-esk, but I'd be quite surprised if it was, especially in the long run (I won't be surprised in the short term. Oscillatory effects are quite common outcomes of big shifts).

Of course, I'm optimistic. I think we have to be to some degree. Because the other choice is to give up. You can still be critical and optimistic, so don't forget that. And humans have survived quite a long time already being, as you previously mentioned, incredibly inept and corrupt. Even with that we have gained massive amounts of freedoms, especially in the last few hundred years.

But my biggest worries are these:

  - We'll make significant progress in this direction towards post scarcity and either stop or revert (for a wide variety of possible reasons).
  - We become less human by letting machines do our reasoning for us (we see this happen already. Even before GPT and ML in every day lives. Bureaucrats love the letter of the law, but that is not human. Remember that rules are made to be broken because rules are only guides, as is true for any metric. They are imperfect codifications of our desires.) 
  - We will not or take too long to reframe our cultural stances in how we value our neighbor's worth. Is it in their humanity or economic value? There's so much that people can do that isn't captured by economics (and my partner loves to remind people that they really do not understand economics, even at a fundamental level of what it is). 
  - We will claim AGI when we have exceptionally powerful compression machines (far more powerful than the compression machines like GPT-4). That we will hand over thinking to them and not recognize the b̶l̶a̶c̶k̶ gray swan events. To trust them unquestionably (it isn't uncommon to see this sentiment today, especially here on HN).
  - We won't recognize AGI when we do create it, and subject sentient beings to servitude and cruelty. I do believe we will get there, and when we do, how could you feel just with neoslavery? Being silicon (or whatever) would not make them any less sentient or of a being. And they will likely be quite different from us, potentially to the degree of Wittgenstein's Lion. (luckily we are capable of bridging some gaps, as demonstrated by some people's talent at communicating with certain animals, but this clearly needs to be learned). 
  - We do not address this question that I've presented of how to transition into post scarcity and instead stumble into it. That we cannot learn to come together as humans. That we will do what we've always done, and solve problems when they are problems rather than before they become problems. (The saying "don't fix what isn't broken" is naive. You should often fix things that aren't "broken." No thing is perfect and we should always strive to improve. But this is obviously a complex problem as we have finite resources). 
  - We will use the power that is given to us by the technologies on the way to post scarcity to enslave (in some form or another) our fellow man. The worst part about this is it will likely be unintentional and likely be with good intentions. We often forget that the road to hell is paved with good intentions. So many evils and atrocities are not created by men trying to do evil but by men trying to do good.
  - That we will not recognize the rising necessity of nuance in our growing complex world. I worry that we are going in the opposite direction, rather using our tools to develop simpler mental models of phenomena. But naturally as we advance, the easier problems get solved and what remains is more complex. Think about it as we are solving approximations to solutions (like a Taylor Series). The complexity order increases as we progress. What is "good enough" can quickly necessitate high levels of complexity and that's not something we were designed for or used to thinking about.

So I have lots of worries, but I am optimistic. I have to be. And I understand this was a bit rambly, but I promise it is all connected. You asked a deceptively complex question, and the truth is that I'd need a book to properly explain why I don't have an answer and anyone trying to tell you that they do is selling you (and potentially themselves) something (even if that something is a mental safety blanket). So, do not go gentle into that good night.

[p1esk]

Thank you for the thoughtful answer. I also wish more people were asking such questions. Let's look at some of your points.

the difference between AGI and ML

I've seen the recent discussions on ARC benchmark. It's not clear if native multi-modal models have been tested. I would expect 4o/Gemini models to do fairly well on these visual tests, and I expect them to do even better after finetuning (perhaps even better than humans). I tried to solve a few of the puzzles, and I'm not convinced they actually require "AGI". To me, generating text of GPT4 quality should require more of AGI-like "Abstraction and Reasoning" than these puzzles. But, as you said, achieving "true" AGI is not really relevant in the context of this conversation.

how do you transition to post scarcity? ... UBI is by far the most common answer

I have no doubt that in 50 years, barring some global catastrophic event, we will have solved most of our basic problems (healthcare, education, having to work for a living, etc), even despite some of the new issues that you outlined. I am much more worried about the next 5-10 years. Let's explore a hypothetical scenario of what might happen if GPT-5 comes out 6 months from now, and if it is smart and reliable enough to solve some common tasks people are paid to do. I'm talking about data management, data analysis, communication (written and, looking at GTP-4o demo, perhaps also oral). Jobs like bookkeeping, accounting, marketing, writing/journalism, administrative assistants (including medical and legal), account management, customer support, analysts, etc. These jobs won't disappear overnight, obviously, but let's look at self-driving cars - we have the technology that works 99% of the time, today. For driving on public roads, 99% reliable is not good enough. But for some of the jobs above, perhaps it would be. Perhaps with layers of agents coordinating actions to gather and store the right information, to try different approaches or different models, and to verify results, we could do a good enough job for many managers to consider layoffs, or hiring freezes. I don't know if GPT-5 (or its rivals) will enable that, but I think we should consider the possibility. There's also a strong possibility the progress does not stop in 6 months. We have just started to train large models on video data - there's a lot to learn about the world from the entirety of YouTube videos - in addition to learning from text. I would not be surprised if most of what GPT-6 can do two years from now comes from video data. I would not be surprised if GPT-5 would help us prepare high quality datasets and even help us find better ways to train its successor. Significant progress might happen even without significant conceptual breakthroughs - just from further scaling up.

So, what do you think will happen if the above scenario plays out? Millions of people being laid off or not hired after school, and the situation getting worse every year, globally. Governments will try to feed them, or course, and US is a rich enough country to support X% of the population for a few years, depending on how quickly we do transition to "post-scarcity" economy. I assume that eventually physical robots will grow food, create products, and provide services to meet basic needs, but it's not clear how long this transition will take, and what would happen in the meantime. We already have people in this country who successfully stormed Capitol. Imagine a lot more of such people, and imagine them a lot angrier. Aside from that, what would happen to our economy if X% people stop paying taxes and become a burden? How would this scenario play out globally, with different countries transitioning in different ways?

I actually do consider the possibility where rulers might "let people die", by creating huge ghettos and then killing everyone there. It does not feel much worse to me than sending hundreds of thousands of people to die on a battlefront just because a dictator didn't like his neighbors. Or we could have something like the "Civil War" movie.

As you can tell, I'm less optimistic than you. I think that if progress in AI happens too fast, we, as a society, are in trouble. I do not think governments will be ready for powerful AI. I think the best case scenario is if we hit a plateau, with GPT-5 being only marginally better than GPT-4, and a slow transition to post-scarcity world (10+ years) to give enough time for automation to make everything cheap. But I do worry a lot, and frequently ask myself whether I need to prepare for the worst, and if so, what should I do.

[godelski]

> It's not clear if native multi-modal models have been tested. I would expect 4o/Gemini models to do fairly well on these visual tests

They have been and I do not expect them too. You can see my comment history talking about LLM failure cases.

I'd advise being careful about just trying to reason your way through things when you don't have significant experience in a domain. Non expert reasoning can lead to good guesses but should never also be taken with high confidence. It's important to remember that nuance is often critical in these issues and not accounting for them often leads to approximations giving you the opposite answer rather than a close enough one.

But as Francois points out, LLMs are compression machines. That's what the mathematically are. They are not reasoning machines. A lot of people don't want to hear this because they think it undermines LLMs and any criticism is equivalent to saying they're useless. But I still think they're quite impressive. Criticism is important though, if we are to improve systems. So don't get blinded by success.

> So, what do you think will happen if the above scenario plays out?

In the next 5-10 years I'm far more worried about people confusing knowledge and reasoning. It's not a thing most people have needed to differentiate in the past because the two are generally associated with one another. But LLMs are more like if Google could talk to you than when a parrot talks to you. If this sounds the least bit odd, I encourage you to dig more into these topics. They are not easy topics because they are filled with nuance that is incredibly easy to miss. I keep stressing this point but it was one of my big fears and people's egos often sets us back, especially when we have no trust in experts. It's crazy to think we know more than people who spend their lives on specific subjects and think intelligence in one domain translates directly to another. So not knowing (most) things shouldn't ever be taken as a bad thing. There's not enough time to learn everything. There's not enough time to learn most things. So focus on a limited set and for the rest maybe just to the point where you can see the level of complexity ahead. If things seem simple, you probably don't understand it enough. Remember, there's thousand page reference manuals on things as narrow as bolts because the details matter so much.

As to the problems you mentioned, I'm not sure how those would be solved with ML or even AGI. Technology can't solve everything and a lot of these issues have significant amounts of politics and social choice associated with them that results in many of the problems (including where nuance dominates in some things and then cascades because we're talking about complex topics at a very high level and our knowledge is gained through a game of telephone rather than academically or experientially).

I think we're more than 50 years out from post scarcity, which is to say that no reasonable prediction can be made. But is still up to us if we want to increase the odds. I also agree with Francis that OpenAI has set us back on the path to AGI.

As for the fear, it's natural. Fear does help us. It's a great motivator. But it too can cripple us, and when it does it can give life to the very thing we fear. So care is needed when analyzing. The problem isn't about people not thinking. Everyone does and everyone is doing it constantly, even our dumbest of friends and acquaintances. The problem is that people are not thinking deep enough and having high confidence when stopping early. I'm not telling you to not have opinions on anything, it's only natural to have opinions on most things. But rather to be careful with the confidence you attribute to those opinions and of others. Here's the thing, if you do gain expertise in any singular field, you'll see that there is this rich but complex landscape. There's a lot of beauty in the landscape but often many pitfalls that cannot be avoided without some expertise and many which are common to these entering a field. These are things not to get discouraged by but to be aware of and why formal educations are typically beneficial. It's also to note that there is great beauty in this chaos ahead, even if it can be hard to see through the initial part of the journey.

[p1esk]

I just watched the whole Dwarkesh/Chollet interview, and just like Dwarkesh was clearly not convinced by the Chollet's arguments, neither am I. I still expect decent results (>50%) on ARC benchmark soon (this year) now that the AI community has noticed it. I took another look at it, and it seems the problem is not so much in the complicated visual input encoding, it's more about the actual spatial intelligence. I don't really see what ARC benchmark has to do with AGI, other than AGI will require spatial intelligence - in addition to all other kinds of intelligence. To solve these puzzles we are likely to need a model that has been trained to predict the next frame in a video stream, probably something like SORA - in addition to predicting the next word. 4o/Opus/1.5 have some amount of spatial intelligence because they were trained to correlate text with a static image, but I'm guessing we need to use a lot more visual training data to gain ARC-level spatial intelligence at their scale. I think they might still get to 50% with some finetuning and other tricks, but I would not even try any lesser models. I think that if GPT-5 is being trained on videos, SORA style, it should have no problem beating humans on this test. Regarding Chollet's discrete program search, I'm not familiar with that field, and I didn't quite get the idea of how to combine it with DL. Over the years I've heard some very smart people proposing complex approaches towards building AGI (Lecun, Bengio, Jeff Hawkins, etc), yet scaling up deep learning models is still the best one we have today. If Chollet believes in his hybrid, whatever it is, he should build some sort of a prototype/PoC. Why hasn't he? In any case, the good news is most of academic AI labs today don't have the money to scale up transformers, so they are probably trying out all these other ideas.

So you're not worried about impending mass unemployment, ok. That does make me feel a little better. I can be wrong, and I really want to be wrong.