[janalsncm]

The alignment problem is largely incoherent. Aligned with whom? Most problems will require unequal and arguably unfair sacrifices.

Further, people even obvious and largely inconsequential solutions are controversial. If an AGI suggested wearing a mask during a pandemic that spreads via airborne particles, a large percent of the population would say the AGI is “unaligned” and we’re living through skynet.

[flangola7]

Yes, and those people would be wrong. You're compelling completely right, humanity is not ready to build an AGI. It will not be ready for a long time, if ever. Alignment may not be a thing that can exist.

Overconfident men like Sam Altman are forging ahead with the unjustified hubris that we are ready for this.

[scott-smith_us]

Alignment with human norms and values. Yes, that's not a well-defined thing. But to paraphrase, "I know misalignment when I see it", for example when an AI suggests something like "feeding the homeless to the hungry".

No one is claiming that alignment is a well-defined thing with crisp edges. They're saying that the mechanisms of AIs will favor solutions without regard to any specific constraints we haven't explicitly stated.

[janalsncm]

> Alignment with human norms and values. Yes, that's not a well-defined thing.

What human norms and values? The more I think about this topic, the more I find it to be an impossible task. There is a huge spectrum of norms and values which are held as sacred by some and sacrilege by others.