Read this, from someone who's thought about it a bit more deeply than Mr Yudkowsky.
https://qntm.org/mmacevedo