[olegkikin]

But corporation collections as paperclip maximizers is a bad analogy. Corporations generally produce something large numbers of people want (be it solar panels or cars in his example), whereas a paperclip maximizer AI produces paperclips that only it wants, destroying everything in its path.

[digi_owl]

The "paperclip" in Stross' talk is not the item being produced, but earnings. If a corporations could massively increase its quarterly earnings by starting a nuclear war, it may well decide to do so.

It is not that the item or service is useful or not in the short term, but that it will pursue a set goal with a single minded conviction.

Another analogy may well be the Sorcerer's Apprentice.

[Veedrac]

The risk of a paperclip maximiser isn't that the paperclip maximiser wants paperclips, but that the limits of almost any reasonable goal become strongly perverse when one approaches the limits of feasible computation.

Corporations are different because there are very strong diminishing returns on scale when it comes to how smart they can be.