[cs02rm0]

I've only had a quick look. I really like the idea of the site, almost desperately so. But the lists of top things to work on that I found seemed a bit disappointing. AI safety policy made the top of one list.

My undergrad AI lecturer told us that it was widely "known" AI was soon going to be generally smarter than humans when he was an undergrad. He wasn't convinced evolutionary changes were going to get us there. It was some 15 years ago I was listening to him, probably 20 or more years before that when he was an undergrad. We still seem to be saying it's 10, 20, or some other made up number of years away.

I'm not even sure there's a significant future problem here. Not on the scale of, say, becoming a multi-planetary species or resolving issues with antibiotic resistance.

[dogcomplex]

Fair enough, but consider that a "dumb" AI running the world on incentives that don't match human and planet welfare is already our current condition - it's called The Economy. While the overall system isn't a perfect AI just yet, any specializable part is already operating at an intelligence far past the point of (most) individual human ability, and the tools to control this system (monetary policy, democratic systems) are increasingly being eroded (untraceable/uncontrollable money systems, pay-to-win democracy). There may be some human intelligence amongst the overall system but, even if we never make a Perfect AI to rule it all, the point at which we're not able to really change the incentive/value system is coming near - if it hasn't passed already.

(That probably has nothing to do with the actual AI Safety problems studied today in regards to pure AIs. But damned if this isn't an important-A-F cousin of the problem.)

[jarofgreen]

Is that not talking about general A.I. tho? And sure, many people argue that might never happen ... But we already have many specific A.I.s, and those specific A.I.s are already making lots of big decisions that affect people's lifes. Sounds like some safety policy there could be worth thinking about.

[ndnxhs]

Its not particularly interesting though. A calculator is a kind of specilized AI that can do maths massively faster than a human can.

[jarofgreen]

Well, it's not going to be interesting to everyone. But there are many people who are interested in it; Cathy O'Neil wrote a whole book about it. The main point is I'd argue that it's a big problem that needs people to look at it. (80,000 hours website was actually talking about general AI - https://80000hours.org/career-guide/world-problems/ - but I'd still argue specific A.I.s need attention too)

[cs02rm0]

Yes. But then specific AI safety is an even smaller problem.

[jarofgreen]

Disagree. It's a different problem, not necessarily smaller. It's clearly not yet a solved problem, and as some of the issues interact with many other social issues there are no easy answers. https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction is the usual starting point referenced here.