[utucuro]

To be frank, I'm not afraid of AI, I'm afraid of the exact opposite - natural stupidity. This story clearly shows that too - "algorithm trained to prioritize scoring points tries to do so at any cost" It's not even emergent behavior, it's badly defined goals...

[arisAlexis]

Read up about the "alignment problem". It is one that ai researchers cannot Crack and its literally what you wrote. The idea that we could never in any way define goals align with us.

[rbanffy]

We have been living with this problem of poorly defined goals for ages. Computers do what they are told to, not what we want them to. I built a career on translating what people want computers to do into what computers need to be told.

I suspect we'll hear of Dr. Susan Calvin more and more in the future.

[utucuro]

I already know about the efforts about AI alignment - this is the reason I'm afraid about natural stupidity. I really do not want to see the alignment issue solved in my lifetime.

[arisAlexis]

Cringe comment: the reason why will not see it both you and me will be the same if it doesn't get solved :)