[Misdicorl]

I think you have this backwards. The missing magic is consequences. From there, AIs that avoid bad consequences (reduced compute budget?) and seek good consequences (??) may develop a 'want' heuristic that favors those things. Or rather, the ones that do will be more successful and the ones that don't will die out.

[pdimitar]

You might be partially correct but you seem to omit built-in wants e.g. hunger or thirst or lust.

Furthermore, if a machine has no wants/needs then it won't take any action at all. It will just stand there and won't even experiment to find the good or bad outcomes.

[ithkuil]

If not doing anything has bad consequences, then it will do something to avoid bad consequences. It seems to me that "want" and "avoid bad consequences" are quite isomorphic

[alexvoda]

It would need "to want" "to avoid bad consequences". It is rather circular because of the word "bad" which implies "do not want".

If it is indifferent to all consequences, you are back to square 1, so no, consequences are not the missing magic. Consequences are a function that transform one "want" into a different "want".

[Misdicorl]

You aren't back to square one. Even indifference will result in selection so long as the indifference is demonstrated in different ways by different AI. The ones which have behavior which accidentally maps slightly better to good outcomes will so better. So long as that gets propagated more strongly to the next generation of AIs than average, you get selection for good outcomes without any explicit 'want' function

[ahefner]

Perhaps, while slowly learning, the machine will cry.

[dunefox]

The machine is me.