[theptip]

It’s worrying to see very smart guys like LeCun failing to grok the paper clip maximizer issue (or coffee maximizer as Russell phrases it), which is like the one paragraph summary or elevator pitch for AI risk. I think there are plenty of other valid objections to a high E-risk estimate but that one is non-sensical to me.

I think Robin Hanson has the most cogent objection to high E-risk estimates, which is basically that the chances of a runaway AI are low because if N is the first power level that can self-modify to improve, nation-states (and large corporations) will all have powerful AIs at power level N-1, and so you’d have to “foom” really hard from N to N+10 before anyone else increased power in order to be able to overpower the other non-AGI AIs. So it’s not that we get one crack at getting alignment right; as long as most of the nation-state AIs end up aligned, they should be able to check the unaligned ones.

I can see this resulting in a lot of conflict though, even if it’s not Eleizer’s “kill all humans in a second” scale extinction event. I think it’s quite plausible we’ll see a Butlerian Jihad, less plausible we’ll see an unexpected extinction event from a runaway AGI. Still think it’s worth studying but I’m not convinced we are dramatically underfunding it at this stage.

[ma2rten]

Have you considered that it's not LeCun who is missing something? The AI safety community seems to be unfortunately almost completely separate from the actual AI research community and be making some strong assumptions about how AGI is going to work.

Note that LeCun had a reply in the thread and there was a lot more discussion which GP didn't quote.

[theptip]

Fair, perhaps I should retract “fail to grok” and replace it with “fail to focus on”. It does seem that LeCun understands the objections (though he dismisses them out of hand).

Regardless of who is right or wrong, “Don’t fear the terminator” is a weird straw-man to raise in a discussion about AI risk. He’s setting up a weak opponent to argue against, when the AI risk community have a large repertoire of stronger cases. “Don’t fear the paper clip maximizer” would be a stronger case to put forth IMO.

In his response points 2&3 he asserts that alignment is easy; simply train the AI with laws as part of the objective function and it will never break laws. I think there has been a lot of investigation and discussion as to why this is harder than it sounds. For example LeCun is explicitly talking about current models that are statically trained to a fixed objective function, but one can easily imagine a future agentic AI (imagine “personal Siri) that will continue to grow, learn, and update in the world in response to rewards from its owner. Maybe he is right about near-term models but I’m completely unconvinced that his arguments hold generally.

Anyway, maybe the “terminator scenario” is a concern LeCun hears from uninformed reporters/lay people that he felt the need to debunk. It’s a valid point as far as it goes, but it has little to do with the actual state of the cutting edge of AI risk research.

[nopinsight]

Russell did have good replies to Lecun’s replies.

From my reading of the full article, Bengio who was/is also well-versed in the latest deep learning research was leaning more toward the Russell argument as well.

[nopinsight]

My issue with the Hanson objection as stated above (link to the original would be appreciated) is that it rests on the assumption that the N-1 level AIs still under human control can somehow completely eliminate or suppress the self-modifying AGI long enough until alignment research is complete. Meanwhile, the unaligned AGI could multiply, hide, and accumulate power covertly.

Humanity would also need time to align AGI before any AI reaches the N+10 power level. The existence of all those N-1 level AIs in multiple organizations only means there are more chances of an AGI reaching the critical power level.

[theptip]

I should have taken the time to link it above. This is the Hanson article: https://www.overcomingbias.com/2017/08/foom-justifies-ai-ris...

(It links to a previous debate with Eleizer too.)