What if you were in that situation, but were incorrect about what things are good, and, while you had a better understanding of what actions would result in what results, the 8 year old had a better understanding of what is good?
The 8-year-old might view your intelligent plans as a terrible abuse:
>BUT I WANT TO EAT TEN MORE TWINKIES, YOU'RE SUPPOSED TO SERVE ME WHAT I WANT, NOT DEPRIVE ME
Could we similarly be wrong if we think a future superintelligent AI is abusing us? Should we consider submitting to it?
Hard questions.
What if you were in that situation, but were incorrect about what things are good, and, while you had a better understanding of what actions would result in what results, the 8 year old had a better understanding of what is good?