[card_zero]

Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?

[stoniejohnson]

I didn't mention it but I fully agree, I imagine ASI would have be to embodied.

My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.

Regarding

> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.

Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.

Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.

[card_zero]

Smarts = ideas, and the available ideas are ours, which contain our values. Where's it going to learn its morality from?

* No values at all = no motivation to even move.

* Some ad hoc home-spun jungle morality like a feral child - in that case, it would lack heuristics in general and wouldn't be so smart. Even your artificial super-brain has to do the "standing on the shoulders of giants" thing.

* It gets its moral ideas from a malevolent alien or axe murderer - how come? Unless it was personally trained and nurtured by Mark Zuckerberg I don't see why this would happen.

Mind you, I suppose it's true that even normal humans tend to be somewhat mean and aloof to any outgroup of slightly different humans. So even after it learns our values, there's a definite risk of it being snotty to us.