[weitendorf]

Work on distributed/federated learning. The main doomsday scenario is exactly what Anthropic is enabling.

RSI within a top-down, concentrated power structure creates an unstable equilibrium and will instantly devolve into an epic power struggle. If you control the company/computers/military with the one most powerful thing, you control everything else. It will probably just be seized but then you instantly become a target to all the other dispossessed/scared/opportunistic people who think they would do a better job than you, or just want it. It’s a stupid power fantasy to think that people with guns and will just let you reign from above as some kind of benevolent researcher-king with absolute, unaccountable control over the economy. Even worse if there’s mass joblessness and nothing else keeping most people busy.

If we can build a horizontal, federated (not the performative kind like Bluesky, has to actually be performance competitive and distributed without jank) intelligence on a mix of commodity and specialized hardware - which BTW is exactly what even Anthropic would have to work with too, a bunch of datacenter gpus of different generations + traditional compute + edge compute/network devices, maybe some ASICs, then finally consumer devices and webgpu - then there is much less risk that AI will be used to concentrate power or amplify bad actors (without their actions being immediately reacted against by the overwhelmingly larger set of good or neutral-with-something-to-lose actors).

The main barrier to federated learning is figuring out how to economically structure this, it has to have a self-funding mechanism that is hopefully more grounded in actual value than something like crypto where it’s purely forward-looking demand (/speculation) or artificial/enforced scarcity. Also it obviously has to be secure, but the risks are different vs companies like Anthropic that are trying to guard their IP - in this case it’s mostly just protection from bad actors trying to pollute training data in a way that would only be noticed after it’s expensive to fork away from, plus just generally using it as a malware distribution or data collection mechanism.

[ianm218]

I think this is a cool angle to pursue! But your model of the world definitely has a lot of assumptions here.

> RSI within a top-down, concentrated power structure creates an unstable equilibrium and will instantly devolve into an epic power struggle. If you control the company/computers/military with the one most powerful thing, you control everything else

With a more stable government you can definitely imagine a situation where someone like Anthropic would work closely with the government in a Manhattan style project to solve alignment as soon as RSI is close. This admin definitely has a more "might makes right" view of the world and I'm not sure they would go that direction.