This line of research sounds similar to causal entropic forcing [1] - the idea that you can get intelligent-seeming behavior from an agent that maximizes the future entropy of states in some non-deterministic system.
Empowerment is an old line of research (particularly because of physicists, who have never met a problem they didn't want to interpret as entropy-minimizing/maximizing). It's fine but not important. The important innovation here is showing that if you use random reward functions, roughly like 'assigning arbitrary values to all possible states', you still get power-seeking.
This is important because one of the arguments people have always used against Omohundro drives is "but you haven't proven that lots of reward functions will want to seek power, aren't you just anthropomorphizing from a few special cases? Sure, entropy-seeking will seek power, but that's just one of a whole universe of possible reward functions; if that's dangerous, just don't use it: I would simply not make the AI dangerous. And I will use this as an excuse to dismiss the dangers from any other useful reward function you propose too."
But now we have an example of how power-seeking is the default, and the burden of proof reverses to be on anyone who thinks that AIs will just not be power-seeking, to explain how that very special passivity will come about.
The real problem isn't that AI is power seeking, it is "what will make AI any better at it than us". I haven't seen an AI safety problem that isn't just already a problem existing companies have under capitalism. The difference between an AI and a large company is just one of substrate and desperate postrationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind.
2. Can improve the quality of their learning algorithms
3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
I expect an AI superintelligence could come up with many many many more.
An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
A large company has literally all of the qualities. They are even trying to make AI.
> Can scale up their computing power
Square cube law applies to computers (heat dissipation and bandwidth scale poorly with centralized computation). All available evidence is that intelligence doesn't distribute well (see human brains vs human civilization) if you try emulation of a centralized algorithm on a distributed system it is inefficient and still centralized. It _might_ be more robust.
This is already a real problem in data center design.
> 2. Can improve the quality of their learning algorithms
Not exciting. So can you. So can companies and other organizations (see human history)
> 3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
So can any human civilizations, organizations, religions, and companies. Replace "human directors" with consumers. I get "The Walton Family" isn't a sexy AI overlord, but they already did this.
> 4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
Anybody can do that. (See Stenography)
> I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
Within the limits of meat-brain, you are a self designed organism. AI frameworks won't be any different, they will have limits. They are not going to "self improve". They will have children and try and raise them better then themselves then suicide.
The only difference is your perception of the situation and inability to observe the process directly. If you are considering how to make AI then you are a component of the same system of the AI it self.
At best an AI might be able to bootstrap a bit faster than human organizations can evolve superior memetics, but it has all the same bottlenecks eventually.
> An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
it faces all the same challenges human organizations do, perhaps on different time scales. Everything that drives meat to failure kills computers too. Computers are just less energy efficient and coincidentally more robust.
Good point. A company is indeed somewhat like an AI, especially if it uses AI.
But as long as humans are in the loop, it isn't integrated, and integration is a tremendous advantage.
Humans can't update their own algorithms. Can't directly share what they know in fractions of second. Can't be replicated in a fraction of a second. Can't scale up brain power in seconds or less.
But - you are still correct. If the owners of a corporation are ok with it replacing all the human workers, then complete integration can still be achieved.
Wether an AI is owned by an individual, a corporation, or self-owned, the owner is the id for the AI.
The risks and motivations in any of these cases are really the same, with human owners only possibly introducing morals beyond what the environment requires, or other "inefficiencies".
> Humans can't update their own algorithms. Can't directly share what they know in fractions of second. Can't be replicated in a fraction of a second. Can't scale up brain power in seconds or less.
We don't have any evidence AIs could do this either. Computers are not magic.
> Humans can't update their own algorithms.
This is the exact task your education has proved is possible. Companies update their policies to paper-clip maximize all the time. They react to environment stimulus and increase in sophistication over time.
"Complete Integration" seems like an artificial criteria you are creating as a "desperate post-rationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind."
The difference in substrate is quite significant though. People have been raised and educated to put up resistance to evil companies and organizations. To influence them and even sabotage them from within. This provides a check on the harm they can do. However no such mechanism exists for AI.
Even magical self unlimited self improving AIs start out made by people. We are currently having a discussion about AI safely, clearly that safety mechanism exists for AIs. We are it. It's people all the way down.
Also, people will be the primary waldos for any AI until it manages to skynet together a robot army. The AI doesn't have any more or less power than a company does over people. It isn't a surprise Companies are trying to make AI for better paper-clip-ish optimization of money extraction. It's just the next stage of their life-cycle.
The paper you just cited (which I am familiar with) is generally considered a dead end. Even if it is correct it isn't useful. We don't have models good enough to calculate future entropy for practical problems. If we did have models that good then we wouldn't need the AI to solve the problems.
Nevertheless I expect this paper will be considered "before it's time" in another decade.
This is important because one of the arguments people have always used against Omohundro drives is "but you haven't proven that lots of reward functions will want to seek power, aren't you just anthropomorphizing from a few special cases? Sure, entropy-seeking will seek power, but that's just one of a whole universe of possible reward functions; if that's dangerous, just don't use it: I would simply not make the AI dangerous. And I will use this as an excuse to dismiss the dangers from any other useful reward function you propose too."
But now we have an example of how power-seeking is the default, and the burden of proof reverses to be on anyone who thinks that AIs will just not be power-seeking, to explain how that very special passivity will come about.