What immediately follows is rather sobering, too: "No machines with self-sustaining long-term goals and intent have been developed, nor are they likely to be developed in the near future."
What about corporations? They are self-sustaining non-human information processing systems with long term goals, and they are subject to selection pressure. They are rudimentary humans-in-the-loop artificial life.
I think the most likely route to destructive AI is a corporation with an AI CEO - possibly one that takes over from a human CEO in a boardroom coup.
Corporations already have legal personhood and act in their own interests. It's going to be much easier to automate and formalise business decision making than to develop a true general intelligence with a full spectrum of human characteristics.
This may sound like science fiction, but as competence increases shareholders - who typically are only passingly interested in moral issues - are likely to demand the increased returns an AI CEO can bring.
Many CEOs are already just trying to operate as a share price optimization algorithm. They don't choose to inject human values into their organizations.
> ...operate as a share price optimization algorithm...
That in itself is not so bad, as extremely long-timeframe constraints (say, >50 years) upon such an algorithm could conceivably be consonant with current decision-making behavior that externalizes many input costs (employee overtime, environmental damage, etc.). Running the algorithm to pay out in very short timeframes (a month to a year) due to most CEOs' anticipated short tenure is what seems to cause undesirable optimizations.
I was thinking about a scenario where some crazy billionaire would build an autonomous AI that would operate a fleet of secretive hedge funds and financial trading companies. The AI wouldn't have any need to know what these companies - staffed by real people - are actually doing. It would just try to maximize profit and it would just kill the companies that would deviate too far from the norm, and build another companies in their place. No CEO of any of the companies would ever see this "investor" in person, everything would be sent over email, and done to the billionaire's name. The billionaire would eventually die and those companies would continue to operate completely autonomously.
The paperclip maximizer generally refers to a AGI with a value system that is not aligned with humans. An AGI smart enough to achieve its goals (making paperclips) so efficiently that it becomes a threat to humans through sheer resource consumption.
So it's not a good example of dumb-tiger-AIs occasionally becoming a threat to humans, which still on average are able to outcompete a tiger with ease.
I am thinking about systems set up to protect us can end up hurting us because in order for them to be helpful they get so much power over us that their continued improvements become fundamental to our survival beyond the scope of our ability to understand it.
The "system" does not need to have intent or be even closely aware to be dangerous to humans.
And so the article sets up a false premise if the quoted conclusion is to be the base of judging whether it's going to be a threat to us or not.