[SonOfLilit]

The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.

"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.

They came up with a framework called Coherent Extrapolated Volition to address this specific question. https://en.wikipedia.org/wiki/Coherent_extrapolated_volition

[janalsncm]

Seems like CEV replaces one problem (“what does humanity want?”) with more problems that are probably even harder to answer.

First of all, calling it “coherent” extrapolated volition presupposes that there is such a thing. It doesn’t actually address the objection above, that there may be no such thing. It’s a bit like saying you solved car safety by presupposing a safe car.

Second, it assumes that such a thing can be effectively measured, and there will be no problems or controversies with the extrapolation process itself. There may be several EVs to choose from, and at that point the framework has nothing to say. Maybe we just pick at random then I suppose.

[SonOfLilit]

I hesitated to recommend the CEV paper, because it's written in Yudkowsky's very personal tone, which some enjoy and others find quite abrasive... but then it occurred to me that you asked about philosophy, and I have a book about Lacan nearby (not a book by Lacan, nobody can read that!), and I've peeked at the Tractatus once... Surely, even if you don't like him, Yudkowsky reads like Pratchett in comparison.

So... of course these questions are addressed in the 38 page essay that introduced the idea.

Specifically, it's not "calling it coherent", it's "assigning more importance to the parts that cohere than the parts that diverge" as one of the core principles (it's one philosopher's opinion, others disagree), with a lot of specific guidelines about how to prefer consensus or kicking decisions down the road and how to deal with complications like "what about dolphins" or "what about our great-great-grandchildren who will be as insane in our eyes as we are in the eyes of 17th century westerners, do their 'votes' count too?".

Of course, like any work of philosophy, it presupposes some pretty incredible things (like a Godlike intelligence that can be made to care deeply about following the spirit of this framework). But you could write a worse first draft for "what would we want AI to be aligned to, if we could define to our heart's content?"

https://intelligence.org/files/CEV.pdf