[argonaut]

You're dismissing the paper for not asking the right questions, but you don't propose any questions that you think are better.

> The hard part is actually figuring out what you care about, particularly in the context of a truly universal optimizer that can decide to trade off anything in the pursuit of its objectives.

This seems basically equivalent to what they are saying. A reward function that rewards "what we actually care about." This might seem vague, but that's fine because these are only proposed problems.

[akvadrako]

I'm not sure what point you are trying to make. It's possible to dismiss an idea without providing an alternative. Yes, finding a reward function is equivalent to figuring out what we care about. Both are about as hard as teaching a bacteria to play piano.

The goal is avoiding unsafe AI. The reason such pointless efforts are wasted on this approach is we don't have a good alternative. The only thing I can think of is delaying it's creation indefinitely, but that's also a difficult challenge. For example, in the Dune books, the government outlaws all computers. That might work for a while.

[argonaut]

Let me elaborate. It is easy easy easy to nitpick and find holes in someone's proposals, someone's problem statements, and someone's goals in life.

Statements are adding noise and less than nothing of value if they just consist of telling people they are working on the wrong thing... and not proceeding to tell them what they should instead be working on, and giving clear positive reasons why (instead of negative reasons someone should not be working on something).

Incidentally this is a broader problem with HN discourse.

[stevetrewick]

From my cold, dead hands.

[GuiA]

> You're dismissing the paper for not asking the right questions, but you don't propose any questions that you think are better.

One might wish to point out that the emperor has no clothes and yet have no desire to plan his majesty's outfits for the next 6 months.

[argonaut]

A witty saying proves nothing. See my reply to the other comment.

[fiatmoney]

I'm saying they're modelling the problem incorrectly as a CS endeavor when it has a lot more to do with analytic philosophy.

"How do I make a program make beautiful music" is a CS problem, but only after you have some notion of aesthetics in the first place.

In the context of a universal optimizer, "how do we make this program behave reasonably without bad side effects" is maybe a CS problem, but it's predicated on "how do we codify our notion of reasonable behavior", which is analytic philosophy with probably a bit of social science thrown in.

Problem-posing is itself difficult and how a lot of philosophical breakthroughs are made. If you want rigorous problem-posing where the solution would be handy for AI, hiring a philosopher might be a good start. Very few of us are equipped to do this kind of work, certainly not here in the comments section.