[tarxzvf]

Honestly astonished that superintelligence is a mainstream idea. The story it tells makes sense only if you never bothered to dig further than its surface.

- Replace 'AI' with 'God', does it still make sense?

- Exponents still take time. 2^33 to get to current world population with no hitches.

- Solomonoff / Bekenstein / Gödel - name your favorite limiting theorem.

- For any optimization method we can literally construct a learning problem that it can never successfully learn. Take it a step further and you have a communication channel where the AI listens to everything and understands nothing.

- Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough?

When we get closer to manufacturing universal intelligence its more impressive incarnations will look more like countries and corporations than omnipotent deities. The problems we’ll have to face will have more to do with consciousness and human rights than with alignment. Alignment is really more about automation at the incomprehensible scale, where the clash between dimensionality reduction and Goodhart’s Law becomes absurd.

[edanm]

You're throwing random "objections" with no elaboration, which doesn't prove anything.

> - Replace 'AI' with 'God', does it still make sense?

Maybe in some cases. So what? God is a fairly broad concept in many people's imaginations.

> - Exponents still take time. 2^33 to get to current world population with no hitches.

It's certainly a fairly contested topic in AI safety - how fast will this thing happen? Are we talking seconds? Hours? Months? Years?

There are at least some valid reasons to think it will be on the faster timescale, so not sure saying that exponents take time is a big counterargument.

> - Solomonoff / Bekenstein / Gödel - name your favorite limiting theorem.

I don't think anyone who's well versed in any of these theorems believes that they have anything at all to say about this.

As a simple counterargument - whatever limiting theorem "limits" an AI, can similarly "limit" us.

> - Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough?

I think others have pointed it out, but as compared to the rest of life on the planet, humans are exactly such a force.

[HaukeHi]

> Was any force ever able to get close to world domination?

Evolution? 2.5bn years ago stromatolites changed the atmosphere from a CO2-rich to O2-rich through photosynthesis, because they had no competition. Now plants dominate the earth (≈450 Gt C, the dominant kingdom), then animals (≈2 Gt C, mainly marine, and bacteria (≈70 Gt C) and archaea (≈7 Gt C).

In 2020, global human-made mass exceeded all living biomass ( nature.com/articles/s41586-020-3010-5).

[tarxzvf]

Yes. The only force we know that achieves this is undirected, and no single part of it stays at the top for long. Contrast with superintelligence, a single entity which does not evolve but optimizes in a directed way.

[HaukeHi]

I think evolution is not an undirected process in that sense because it's an optimization process, that optimizes to create more copies of itself. Superintelligence will likely use some Evolutionary Computation (see en.wikipedia.org/wiki/Evolutionary_computation ).

Also see Karl Sims 'Creatures' from the 90s: youtube.com/watch?v=JBgG_VSP7f8 or OpenAI's Multi-Agent Hide and Seek: youtube.com/watch?v=kopoLzvh5jY

[ganzuul]

If it interacts with its environment, it evolves. Environment and genetic evolution are in constant interaction. This is a definition of karma.

[ethbr0]

> Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough?

I'd rebut that most current technologically and financially leading-edge civilizations have optimized for that, at the expense of world domination. Even China has indeed evolved into a more financially (rather than physically) optimized state.

The last time we had major superpowers devoted to total expansionist war was Nazi Germany and Imperial Japan, and neither of them possessed nuclear weapons (or ISTAR, chemical or biological weapons, space capabilities, logistics, or data networks in the modern sense).

If a modern superpower (which is to say, excluding the current Russian government) devoted itself to preparing for and then launching a war of global conquest, who's to say?

I think what does cut against the rebuttal and buttress the "there can never be physical world domination" is the sheer amount of space, relative to a potential aggressor.

There are no world-spanning empires anymore. Consequently, you would have to fight through opponents sequentially, meaning each one is better prepared than the last. And that sounds like futility.

What the Gwern timeline does accurately identify is that the key element is time.

Either everything is over quickly and before opposition begins to mobilize, or there can be no world domination.

[imtringued]

Economists still believe in permanent exponential growth even though the last 2000 years have proved that it doesn't exist.

[Rastonbury]

We haven't reached a limit, no where close, natural resource supply is not limiting growth. While Moore's law is slowing, machines and productivity gains are still being made. Fusion is theoretically possible, when it happens will transform the world