| > Getting superintelligent AI to understand simple specifications should be a non-issue. Why would that be the case? A big part of the worry around AI-alignment is exactly because this seems very hard when you try to do it. We are used to interacting with other humans, who implicitly share almost all our background assumptions when we communicate with them. The same is not the case for a computer program. E.g. if you're holding a basketball and tell you "throw it to me", you implicitly understand that I mean to throw it: 1. To my hands, or to some area that makes it easy to catch it. 2. Throw it slowly enough that it arrives to me. Not strong enough to hurt me. 3. Not to try to bounce it off of something that will break on the way to me, even if it still arrives to me. etc. These are all background assumptions, and I know they're hard to actually specify because smart people have spent twenty years trying to figure out the math to do this and say it's hard. Also, if you think those are contrived exampled - let's note that the closest thing we have to building an AGI right now is just building software, in general. And I think I won't shock anyone by saying that "getting software to do what you want, without bugs" is... hard. I think there are almost literally no software systems today that don't have bugs in them. |