| > You are saying that doing one is 99% of the work and the rest is 1%. Why do you think so? (Different person) I think it's much starker than that, more even than 99.99% to 0.01%; the reason is the curse of high dimensionality. If you imagine a circle, there's a lot of ways to point an arrow that's more than 1.8° away from the x-axis. If you imagine a sphere, there's even more ways to point an arrow that's more than 1.8° away from the x-axis. It gets worse the more dimensions you have, and there's a lot more than two axies of human values; even at a very basic level I can go "oxygen, food, light, heat", and that's living at the level of a battery farmed chicken. Right now, we don't really know how to specify goals for a super-human optimiser well enough to even be sure we'd get all four of those things. Some future Stalin or future Jim Jones might try to make an AGI, "strangle the AGI in its cradle and retry" because they notice it's got one or more of those four wrong, and then finally release an AI that just doesn't care at all about the level of Bis(trifluoromethyl)peroxide in the air, and this future villain don't even know that this is bad for the same reason I just got that name from the Wikipedia "List of highly toxic gases" (because it is not common knowledge): https://en.wikipedia.org/wiki/List_of_highly_toxic_gases |