[pron]

> I just don't understand why people are incapable of extrapolating.

I guess it's a matter of education. People with a mathematics or computer science background know that unless the dynamics of a process are known to be linear, extrapolation is usually wrong. Since we don't know that the dynamics here are linear and have every reason to believe they aren't, extrpolation is unlikely to teach us anything valuable.

[jmoggr]

> and have every reason to believe they aren't

What reason? To me recursive self improvement seems credible, it's just a question of when. It seems obvious that given RSI, trend lines will be at least linear.

The disagreement could just be about if RSI is currently meaningfully happening, but that seems very difficult to tell given the lack of public data.

[pron]

Because most systems in nature and society, including technological progress, aren't linear.

It's certainly plausible that improvements will continue, but the pace is completely unpredictable. It's also plausible that the training material is polluted and progress will not continue. I'm just saying that predicting the rate of technological progress is not easy, and historically, it's rarely been smooth in the long run.

There are many complicating technical factors, but also non-technical ones. Technological improvement in the short term will not necessarily yield commensurate economic gains, in which case, investment may not grow enough to sustain the progress.

As for the recursive part, currently it's either hypothetical or based on too few data points. Not saying it won't happen, but it's far from being the only plausible trajectory.

[jmoggr]

I generally agree with this. I suspect we just have different beliefs about how close we are to RSI.

> Not saying it won't happen, but it's far from being the only plausible trajectory.

But if it does happen, then wouldn't expected outcome be at least linear?

[pron]

> I suspect we just have different beliefs about how close we are to RSI.

I don't have any belief on the matter, but my scepticism isn't necessarily about RSI itself, but about how much it would matter even if it does happen soon. Too many things are limited by lack of resources that no brain in a jar can obtain. And if such a brain in a jar itself is very expensive to operate, it may not be easy for it to justify its existence. My point is that the technical aspect is uncertain, but it is also only a part of a larger system that's has many sources of uncertainty.

[jmoggr]

This makes sense. Interacting with the world is hard and poses real bottlenecks.

I think that resource acquisition is a solvable problem for such a system, robots seem mainly limited by software.

I concede that I am too certain about the linearity of it. But like, cmon man, in the limit entropy ensures that everything ends. The world could still get very weird very fast. I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.

That's all I'm trying to argue at least. And I admit that waving my hand at linearity sounds uneducated.

[pron]

> robots seem mainly limited by software

First, I don't think so. Second, some resources would take even robots decades or centuries to collect. Even something as fundamental as energy production takes a lot of time and money to build. The question of where we'll get the energy to run the robots is not a simple one, and it's one of the simplest questions involved.

> I think that's what original comment was trying to argue, and I think it's possible within the limits you've laid out.

Possible isn't the same as likely, and the reason we don't extrapolate is that different extrapolations lead to very different results.

But regardless of what happens and when, I think that we, people educated in computer science, should remember that many questions are simply not answerable in a short amount of time, and we know with absolute certainty that answering them is not a question of intelligence but of computational power and time. We no that no human or machine, however intelligent, can predict or control the nonlinear systems that are all around us because they are computationally intractable.

[apsurd]

no because we have no way to understand the full consequence chain of any stated goal. We see this time and time again in all complex system goals; unintended consequences.

So RSI toward what prompted goal? Then, how can you be so sure that reaching that goal looks like the straight line you envision?

edit: more accurately, it's about incentives. We generally understand that encoding a goal in a complex system is about aligning incentives. So now it's the incentives that are proxies for intended outcomes. But as mentioned, we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes. And it makes sense because complex systems are not a straight line.

[jmoggr]

Is a coherent, human-interpretable, goal necessary for recursive improvement?

> how can you be so sure that reaching that goal looks like the straight line you envision?

Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI?

> we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes.

We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction (with unintended outcomes). I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.

[pron]

> Because the thing pursuing the goal would be able to improve it's ability to pursue the goal

But not necessarily without needing a huge amount of resources. It is a mathematical certainty that no intelligence can solve computationally intractable problems (including forecasting the weather, or the economy) without access to resources we simply don't have.

[cheevly]

There are at least a dozen scaling laws for AI right now. It’s improving on so many different fronts that people don’t seem to comprehend.

[pron]

We comprehend, but these "scaling laws" have been in effect for less than a decade (and they're more historical observations over a very short period than actual laws), and while some technologies progress exponentially for some amount of time, the complexity of some computational problems grows exponentially forever. For example, if computational resources double every year, it may still be five centuries before some computational problems can become practically computable. It is mathematically proven that no amount of intelligence can compensate for resources, and even if exponential growth could be sustained for a long time, and that's a big if, an exponential curve still grows slowly at the beginning.

For example, suppose AI helps us figure out a way to exploit much more of the sun's energy. Accomplishing that necessary preliminary can, on its own, take many decades, and if we split our efforts among multiple approaches, it may take longer. Intelligence can't break the actual laws of mathematics or of physics. And that's before we consider things like how resources will be allocated when AI tells people that man-made climate change and transgenderism are real.

This generation of AI hasn't even hit its first crisis. Assuming there will be none is like settlers assuming their town will never suffer a major earthquake because they haven't had one in five years.

Of course, improvements in problems that may not grow exponentially also matter a great deal, but there are too many unknowns (look at how many unknowns there are around quantum computing).