| No. Sorry, but this is wrong and Yudkowsky is both naïve and mostly exists in the domain of fan fiction. There are way way way too many issues that are addressed with a hand-wave around scenarios like “AI developing super intelligence in secret and spreading itself around decentralized computers while getting forever smarter by reading the internet.” Too many of his arguments depend on stealth for systems that take up datacenters and need whole-city-block scales of power to operate. Physically, it’s just not possible to move these things. Economically, it wouldn’t be either close to plausible either. Power is expensive and it is the single most monitored element in every datacenter. There is nothing large that happens in a datacenter that is not monitored. There is nothing that is going to train on meaningful datasets in secret. “What about as technology increases and we get more efficient at computing?” We use more power for computing every year, not less. Yes, we get more efficient, but we don’t get less energy intensive. We don’t substitute computing capacity. We add more. The same old servers are still there contributing to the compute. Google has data centers filled with old shit. That’s why the standard core in GCP is 2 GHz. Sometimes, your project gets put on a box from 2010, other times, it gets put on a box from 2023. That process is abstracted so you can’t tell the difference. TLDR: Yudkowsky’s arguments are merely fan fiction. People don’t understand ML systems so they imagine an end state without understanding how to get there. These are the same people who imagine Light Speed Steam Trains. “We need faster rail service, so let’s just keep adding steam to our steam trains and accelerate to the speed of light.” That’s exactly what these AI Doom arguments sound like to people in the field. Your AI Doom scenario, although it might be very imaginative, is a Light Speed Steam Train. It falls apart the moment you try to trace a path from today to doom. |