[bilater]

I just don't understand why people are incapable of extrapolating. AI can't do this blah blah, my brother in spaghetti monster, have you seen the curve? ChatGPT launched in November, 2022. Opus in December last year. Do you see where this is going?

AI may be a tool and add 50 IQ points to a 100 IQ person today but what happens when its adding 10000 IQ points. The 100 base doesn't quite matter as much does it? What does that world look like?

No, it's not just next word prediction. No you can't just wave your hand that it's probabilistic, so insert bad analogy to compiler or calculator and go about your day. And no, you can't just look at its limitations today, make a conclusion, and go back under your rock.

[encomiast]

> ChatGPT launched in November, 2022. Opus in December last year. Do you see where this is going?

Maybe this is tongue-in-cheek, but in case it's not.

No, nobody does. Consider that a person in their lifetime could have seen the Wright Brothers first flight in 1903, then the first jet engine in 1939, then commercial jet travel in the 1950s. That is an amazing 50 years for air travel. If you were living in 1950 and were to extrapolate where air travel would be by 2026, you would think we would be taking routine trips to Mars. Instead what we got was TSA, cramped seats, and better safety. But "progress" leveled off dramatically. Commercial air flight has looked pretty much the same for my entire life.

We have no idea how far transformers and LLMs will take us. It certainly isn't obvious where this is going.

[jmoggr]

I think it's credible to believe that that a system which is able to meaningfully improve it's own intelligence will lead to a far weirder world than the one imagined by the article.

The parent comment could be interpreted as frustration from a lack of imagination about how weird the future could be.

Perhaps we're not at the recursive self improvement yet, but it seems increasingly naive to believe that it isn't possible.

[encomiast]

Of course you might be right. But if we look at our past ability to predict the future by extrapolating from the present and recent past, you might also say it is naive to think that _this time_ is somehow different, that this time the future is clear.

Also, I think you've set up a straw man. I haven't seen anyone arguing that this future isn't possible. What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.

[bilater]

I agree with you that there is no certainty, and in fact the future will almost certainly not look like anything we imagine. I think that's what makes it exciting.

We think about starships and humans mining Mars, and it'll probably be nothing like that. It might just be humanoids or AI drones that actually end up colonizing the solar system, with the humans never really leaving earth except as mind uploads...

But at the end of the day, things are changing. Things are accelerating. And our mental models of what works or doesn't work will almost certainly break. I don't fear the future, nor do I project any confidence on what that's going to look like, but I know I want to be part of it.

[jmoggr]

> you've set up a straw man

I think you're right, my bad.

> What I see is resistance to the idea that we can have any certainty about the future by drawing a straight line from the past.

I'm arguing that if recursive self improvement happens, that the trend line will be reasonably predictable. (and that we are close enough to RSI that we should take this possibility seriously).

[pron]

> I just don't understand why people are incapable of extrapolating.

I guess it's a matter of education. People with a mathematics or computer science background know that unless the dynamics of a process are known to be linear, extrapolation is usually wrong. Since we don't know that the dynamics here are linear and have every reason to believe they aren't, extrpolation is unlikely to teach us anything valuable.

[jmoggr]

> and have every reason to believe they aren't

What reason? To me recursive self improvement seems credible, it's just a question of when. It seems obvious that given RSI, trend lines will be at least linear.

The disagreement could just be about if RSI is currently meaningfully happening, but that seems very difficult to tell given the lack of public data.

[pron]

Because most systems in nature and society, including technological progress, aren't linear.

It's certainly plausible that improvements will continue, but the pace is completely unpredictable. It's also plausible that the training material is polluted and progress will not continue. I'm just saying that predicting the rate of technological progress is not easy, and historically, it's rarely been smooth in the long run.

There are many complicating technical factors, but also non-technical ones. Technological improvement in the short term will not necessarily yield commensurate economic gains, in which case, investment may not grow enough to sustain the progress.

As for the recursive part, currently it's either hypothetical or based on too few data points. Not saying it won't happen, but it's far from being the only plausible trajectory.

[jmoggr]

I generally agree with this. I suspect we just have different beliefs about how close we are to RSI.

> Not saying it won't happen, but it's far from being the only plausible trajectory.

But if it does happen, then wouldn't expected outcome be at least linear?

[pron]

> I suspect we just have different beliefs about how close we are to RSI.

I don't have any belief on the matter, but my scepticism isn't necessarily about RSI itself, but about how much it would matter even if it does happen soon. Too many things are limited by lack of resources that no brain in a jar can obtain. And if such a brain in a jar itself is very expensive to operate, it may not be easy for it to justify its existence. My point is that the technical aspect is uncertain, but it is also only a part of a larger system that's has many sources of uncertainty.

[jmoggr]

This makes sense. Interacting with the world is hard and poses real bottlenecks.

I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.

I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast. I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.

That's all I'm trying to argue at least. And I admit that waving my hand at linearity sounds uneducated.

[apsurd]

no because we have no way to understand the full consequence chain of any stated goal. We see this time and time again in all complex system goals; unintended consequences.

So RSI toward what prompted goal? Then, how can you be so sure that reaching that goal looks like the straight line you envision?

edit: more accurately, it's about incentives. We generally understand that encoding a goal in a complex system is about aligning incentives. So now it's the incentives that are proxies for intended outcomes. But as mentioned, we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes. And it makes sense because complex systems are not a straight line.

[jmoggr]

Is a coherent, human-interpretable, goal necessary for recursive improvement?

> how can you be so sure that reaching that goal looks like the straight line you envision?

Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?

> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.

We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction (with unintended outcomes). I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.

[cheevly]

There are at least a dozen scaling laws for AI right now. It’s improving on so many different fronts that people don’t seem to comprehend.

[pron]

We comprehend, but these "scaling laws" have been in effect for less than a decade (and they're more historical observations over a very short period than actual laws), and while some technologies progress exponentially for some amount of time, the complexity of some computational problems grows exponentially forever. For example, if computational resources double every year, it may still be five centuries before some computational problems can become practically computable. It is mathematically proven that no amount of intelligence can compensate for resources, and even if exponential growth could be sustained for a long time, and that's a big if, an exponential curve still grows slowly at the beginning.

For example, suppose AI helps us figure out a way to exploit much more of the sun's energy. Accomplishing that necessary preliminary can, on its own, take many decades, and if we split our efforts among multiple approaches, it may take longer. Intelligence can't break the actual laws of mathematics or of physics. And that's before we consider things like how resources will be allocated when AI tells people that man-made climate change and transgenderism are real.

This generation of AI hasn't even hit its first crisis. Assuming there will be none is like settlers assuming their town will never suffer a major earthquake because they haven't had one in five years.

Of course, improvements in problems that may not grow exponentially also matter a great deal, but there are too many unknowns (look at how many unknowns there are around quantum computing).

[apsurd]

but you’re just on the other end doing the same thing.

one end is luddites nothing to see here, and the other end is omg humanity is cooked because programming is solved! and consciousness is coding so -> gg

yes there’s a wide surface of emergent behavior we haven’t fully explored, but why do you get the pass to draw a straight line to a conclusion? the naysayers are doing the same thing.

[bilater]

At least I have trends backing up my claim. What precedent is there for saying, "Yeah, AI has kept getting better, and keeps doing things that we thought it would never be able to do, but now I think it’s going to plateau because I just can’t see the model writing good poetry or being smart enough to make determinsitic tool calls" or whatever?

[skydhash]

Most people are not against AI getting better per se. They’re about AI boosters not able to prove their outlandish claims.

We’ve seen a lot of claims over the years substituting one metric (amount of code written) for another (useful code produced) to justify that AI are getting “better”. We just want a proper definition of better.

[Capricorn2481]

> At least I have trends backing up my claim

No, you don't have trends backing your claims that we are going to add 10,000 IQ points to the average person. Anymore than the S&P 500 is going to 100x in a decade years.

[apsurd]

it doesn’t have to be that deep or specific.

all things decay on a curve. pretty much all things in life are not modeled by a straight line.

so you’re saying no theres still more UP. sure, but that’s a tale as old as time. when where and how much in how long is the reality piece.

[bilater]

The line is not just AI. It’s our whole story. You can extrapolate it to Moore’s law, automation, agriculture, and so on. And there's no decay. It's just perception.

We are already in the singularity. We’ve always been. It just gets faster at each step.

[apsurd]

I've read Sapiens. And I liked it. It's very insightful. But that perception is a perception so I don't understand the distinction you're making.

Human advancement is a story we tell ourselves. Agree. That it in any way looks like a straight line is just not true. If nothing else, it looks coherent after the fact... because it's an extremely lossy story our limited brains can comprehend. But it's not reality. No entity in the known Universe "comprehends" the march of progress. They weren't there when it started. So I just don't know what we're talking about anymore.

[rpdillon]

We still have far to go. You can tell by the pace if improvement in so many dimensions. If the popcorn's popping had started slowing down, I might believe you, but instead it's speeding up.

[cj]

I’m not convinced model productivity will scale forever.

Presumably there are a lot of use cases that are essentially “solved” where more powerful models won’t matter.

[the__alchemist]

I encourage you to do research on extrapolation as a general concept. Not a deep dive, but to explore its general utility and limitations as a tool. I also encourage you to explore what metrics you use when evaluating the utility and impacts of LLMs.

[bilater]

I encourage you to use your imagination once in a while.

[ustad]

“ChatGPT launched in November, 2022. Opus in December last year.”

You seem to be implying that there was nothing before. Obviously not true.

“AI may be a tool and add 50 IQ points to a 100 IQ person today but what happens when its adding 10000 IQ points.”

What is even IQ? What if it adds 50 now, 25 next month, 12.5 the month after, etc. You should check out Zeno.

[lillutoo]

Extrapolation leaves you in a world with no agency. Okay, so the models become superintelligent, you're kind of fucked at that point and there's no value you as a human can add so that reality isn't really productive to think about.

The only thing we can do is look at the current state of things, and we still have the same problems we've always had with these models, they need proper direction and input.

[bilater]

That is the question, and it does f** with me. But the existential nihilism that comes with it is not an argument against it not happening.

[Rendello]

I've been thinking about this quote a lot:

> [...] we are living in a society [that is out] virtually to satisfy and gratify each and every human need, except for one need, the most basic and fundamental need operant in man, the need for meaning.

https://youtu.be/GTbliwS0gS4?t=153

Maybe the Amish had it right, after all.

[bilater]

I reject the premise that we need to go back to some old ways of living to find meaning. My life is full of meaning, but it would probably look like meaningless abundance to someone who was toiling on a farm 18 hours a day or being chased by saber-toothed tigers.

The next paradigm will be similar. It might seem meaningless to us. For example, most of our identity is tied to our jobs in our current society. This will have to change. What does that look like? Do we create content and share our art? Do followers and attention become the new currency?

[apsurd]

I don’t think meaning is as malleable as implied. Yes, meaning is entirely subjective, but it’s not boundlessly so. The reason history is a guide is because the needs were more directly connected to the whole maslov’s hierarchy thing. Survival is more justifiably meaningful than “being famous”. I don’t think seeking attention is at all justifiably meaningful in regards to this proposed future attention economy.

“Man’s search for meaning” is arguably the most visceral human framed definition of meaning. Let’s just say it’s not about attention and the Metaverse.

[Terretta]

Extrapolate the flight time from NYC to London in 1976 ... have you seen the J-curve?

Or extrapolate the curve for self driving cars once they were only "two years away".

Catch is, tech progress is not one curve. It's overlapping S curves, stair steps of progress and stagnation, with the next iteration starting off worse than the current one, until crossing it as the current one tops out.

[cadamsdotcom]

The problem with your post is you’re depending on an assumption that all types of intelligence are useful in every situation.

Being embodied will always be a huge advantage over a “country full of geniuses in a datacenter”.

[AngryData]

Is there demand for 10000 iq intellegence to fund its construction if we already have +100 iq AI? You are assuming there will be demand that scales as fast as technology can to fund its continued fast pace development.

[jmoggr]

> Is there demand for 10000 iq intellegence

Yes. If it looks like 10000 iq is possible, then money will be thrown at whoever is most likely to achieve it first, and whoever has the most intelligent models right now strongly predicts who will be first.

This is driven by the belief that whoever gets to 10000 iq first will likely dominate the majority of all economic activity, since it will be more efficient to trade with them than with some less efficient (dumber).

This does not depend on traditional economic demand, at some point this becomes a self-fulfilling prophecy.

[AngryData]

People with money to invest will throw money at it to build it, but who is the customer base after that? So one guy eventually controls the stock market, why would I as a human want anything to do with that market? Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid? Unless I start at the top, ill never really go meaningfully higher.

Instead I can spend far less to use my 200 iq machine that already exceeds everything I could ever want. Im not ever going to personally fund a war of intellegence against other intellegences so anything more is of marginal value. The same reason I don't own a 2,000 horsepower truck or tractor, there are almost no uses for me to want it.

[jmoggr]

> but who is the customer base after that?

That's the fun part, there is none (in the traditional way of people buying things at least).

> Why would I invest my earning into sustaining a machine that ideally is extracting every cent possible without leeway and funneling it up the pyramid.

You shouldn't! Unfortunately everyone acting in their own self interest still results in this getting funded, since if such a machine were to exist, would it not be better to have a share of it?

I agree with your perspective! I just don't think that, as a species, we have a good track record of saying no to the existence of 2000hp trucks with ALMOST no use.

[bilater]

The demand for frontier intelligence is basically infinite. The frontier doesn't just complete a task more effectively, it creates new tasks and new problems and new solutions. It grows the pie exponentially. It's like looking at the internet the first time and trying to imagine the economy that will be built using it: social media, YouTube, Online job boards, Blogs etc.

[hootz]

My nephew grew 7 cm in one year and 10 cm in another, so by 2040 he will be 4 meters tall!

[bilater]

woa your nephew lives inside a height-optimization machine that gets redesigned every few months (weeks), learns from every growth chart on Earth, and keeps proving it can grow faster?

[mythrwy]

The internet doesn't have 10000 IQ points though. So where will it get the knowledge to be that smart?

[_se]

It literally is just next word prediction. If you don't understand that, everything else you're thinking and saying is based on an incorrect foundation.

Take a step back and really take a look at what's going on. You are not thinking clearly.

[bilater]

https://www.astralcodexten.com/p/next-token-predictor-is-an-...

[naveen99]

That’s not how iq works.