[tiborsaas]

If we are speculating on existential risks, then consider Satya Nadella's take on the future of software: https://www.youtube.com/watch?v=a_RjOhCkhvQ

It's quite creepy that in his view all tools, features, access and capabilities will be accessible to an AI agent which can just do the task. This sounds fine in a narrow scope, but if it's deployed at the scale of Windows, then it suddenly becomes a lot more scary. Don't just think of the home users, but businesses and institutions will be running these systems.

The core problem is that we can't be sure what a new generation of AI models will be capable of after a few years of iteration. They might find it trivial to control our machines which can provide them unprecedented access to follow an agenda. Malware exists today to do this, but they can be spotted, isolated and analyzed. When the OS by design is welcoming these attacks there's nothing we can do probably.

But please tell me I've consumed too many sci-fi.