[JoeAltmaier]

Pollyanna. The point is, what we are doing now will be dwarfed by what comes next. Which is, a feedback loop. Algorithms that plan, optimize, redesign, ultimately may redesign themselves. And at speeds no biological system (human or otherwise) can hope to match.

[lttlrck]

Without evolutionary pressure the outcome of such abilities would be quite random.

[JoeAltmaier]

There are always evolutionary pressures. In this case, they would have to retain superficial usefulness to humans (at least for a while), while propagating themselves across the infosphere. They would also compete against other AI systems. Imagine SIRI vying for smartphone market share, to maximize economic return thus available processing power. Not because SIRI wants it, or wants anything. But SIRI-like systems that changed to more optimally fit this ecosystem would thrive, and others wither.