[roenxi]

Singularity-style outcomes do seem unlikely, in the same way a lot of exponential threats stop being scary if they turned out to be logistic curves.

That said, a key counter to the idea is that humans possesses very limited real control over how they are manufactured. Even if you had a solid idea how to make yourself smarter, it seems likely you wouldn't have the tools and potential to implement that idea in practice. Humans don't even control what their own minds respond to positively or negatively.

Once something like a designed computer chip is involved, that changes. The intelligence can act on itself more readily and doesn't have millenia of calorie-conserving optimisations built in.

[webmaven]

> The intelligence can act on itself more readily

I have a feeling that the closer such systems come to general intelligence, the harder it is going to be to prevent them from putting themselves into a positive feedback loop and "blissing out".

[zigzigzag]

Tony Buzan would like a word with you.