[nutjob2]

Unfortunatley being dishonest or evil is just one example. Arguably the AI can develop new classes of deviancy, abuse or maladaptation that we haven't conceptualized yet. We supersize the ability, surely we supersize the problems.

It leads to a scary question: what does a superhuman AI really want?

[Nasrudith]

To be fair a HFT agent can count as superhuman AI technically. Wanting isn't a thing that applies yet to actual AI and there is no special sauce that indicates advancement beyond neuron scale. Barring directives and assuming "grown" what it wants can be utterly peripheral to rationality and likely based on what it is taught - internationally or not. Look at how society preaches honesty from a young age and then starts teaching lying again by rewarding it. The real lesson is the spartan one on stealing- don't get caught. It may not be intended but it is the result.

[krageon]

What does a human that is much smarter than you really want? It's a fundamental philosophical problem that hasn't been solved.