[Houshalter]

Well it is incredibly dangerous, but if we did make safe, super intelligent AI, it would definitely be a much greater investment than directly working on those problems.

To put it lightly, it would automate and massively bring down the cost/increase the speed of research and engineering. Sure enough money, spent on enough humans, given enough time, could eventually find a cure for any disease. But why do that when you can just ask the AI and have a cure overnight?

Of course it's not clear how safe such an AI would be (such a being could easily outsmart us and get what it wants, whatever that even is), nor how difficult it will be to create one.

[throwaway7808]

Well, let's hope they will go broke wasting money into the AI research black hole before being able to actually build one :) Otherwise, you are correct. A successful built of super intelligent AI is probably an extinction event for the humans.

[ra88it]

After super intelligent AI, what's special about humans?

[Houshalter]

Define "special". I don't want to be killed to make way for more paperclips.

[JabavuAdams]

We still have pets, and they're rather well taken care of.

[Houshalter]

That's just another human weakness - we anthropomorphize lesser creatures. And even with that, we still don't look before stepping on ants.

[enupten]

Realistically though, a super intelligent AI, would require far more fundamental breakthroughs. People who say otherwise, should take a look at problems in Control theory.

The effect of scaling things up should be interesting though. Perhaps there is a "Phase transition" like thing, where suddenly something awesome happens. This also means that, sadly, Universities would no longer be able to provide adequate resources for research.

[Houshalter]

What makes you say that? It may seem that it would take a lot of work, but really, how can we know? Often times difficult problems seem obvious in retrospect.