[comp_throw7]

I think this post is sort of confused because, centrally, the reason "AI Alignment" is a thing people talk about is because the problem, as originally envisioned, was to figure out how to avoid having superintelligent AI kill everyone. For a variety of reasons the term no longer refers primarily to that core problem, so the reason so many things that look like engineering problems have that label is mostly a historical artifact.

[godelski]

  > as originally envisioned

This was never the core problem as originally envisioned. This may be the primary problem that the public was first introduced to, but the alignment problem has always been about the gap between intended outcomes and actual outcomes. Goodhart's Law[0].

Super-intelligent AI killing everyone, or even super-dumb AI killing everyone, is a result of the alignment problem when given enough scale. You don't jump to the conclusion of AI killing everyone and post hoc explain through reward hacking, you recognize reward hacking and extrapolate. This is also the reason why it is so important to look at it from engineering problems and from things happening on the smaller scales, *because ignoring all those problems is exactly how you create the scenario of AI killing everyone...*

[0] https://en.wikipedia.org/wiki/Goodhart%27s_law

[Side note] Even look at Asimov and his robot stories. The majority of them are about alignment. His 3 laws were written as things that sound good and have intent that would be clear to any reader, and then he pulls the rug out on you showing how they're naively defined and it isn't so obvious. Kinda like a programmer teaching their kids to make and PB&J Sandwich... https://www.youtube.com/watch?v=FN2RM-CHkuI

[hollerith]

But Asimov never called it alignment: he never used that word or the phrase "aligned with human values". The first people to use that word and that phrase in the context of AI (about 10 to 13 years ago) where concerned mainly with preventing human extinction or something similarly terrible happening after the AI's capability has exceeded human capabilities across all relevant cognitive skills.

BTW, it seems futile to me to try to prevent people from using "AI alignment" in ways not intended by the first people to use it (10 to 13 years ago). A few years ago, writers working for OpenAI started referring to the original concept as "AI superalignment" to distinguish it from newer senses of the phrase, and I will follow that convention here.

>the alignment problem has always been about the gap between intended outcomes and actual outcomes. Goodhart's Law.

Some believe Goodhart captures the essence of the danger; Gordon Seidoh Worley is one such. (I can probably find the URL of a post he wrote a few years ago if you like.) But many of us feel that Eliezer's "coherent extrapolated volition" (CEV) plan published in 2004 would have prevented Goodhart's Law from causing a catastrophe if the CEV plan could have been implemented in time (i.e., before the more reckless AI labs get everyone killed), which looks unlike to many of us now (because there has been so little progress on implementation of the CEV plan in the 21 years since 2004).

The argument that persuaded many of us is that people have a lot of desires, i.e., the algorithmic complexity of human desires is at least dozens or hundreds of bits of information and it is unlikely for that many bits of information to end up in the right place inside the AI by accident or by any process except by human efforts that show much much more mastery of the craft of artificial-mind building than shown by any of the superalignment plans published up to now.

One reply made by many is that we can hope that AI (i.e., AIs too weak to be very dangerous) can help human researchers achieve the necessary mastery, but the problem with that is that the reckless AI researchers have AIs helping them, too, so the fact that AIs can help people design AIs does not ameliorate the main problem: namely, we expect it to prove significantly easier to create a dangerously capable AI than it is to keep a dangerously capable AI aligned with human values, and our main reason for believing that is the rapid progress made on the former concern (especially since the start of the deep-learning revolution in 2006) compared to the painfully slow and very tentative-speculative nature of the progress made on the latter concern since public discussion on the latter concern began in 2002 or so.

[hamburga]

> The argument that persuaded many of us is that people have a lot of desires, i.e., the algorithmic complexity of human desires is at least dozens or hundreds of bits of information

I would really try to disentangle this.

1. I don't know what my desires are. 2. "Desire" itself is a vague word that can't be measured or quantified; where does my desire for "feeling at peace" get encoded in any hypothetical artificial mind? 3. People have different and opposing desires.

Therefore, Coherent Extrapolated Volition is not coherent to me.

This is kind of where I go when I say that any centralized, top-down "grand plan" for AI safety is a folly. On the other hand, we all contribute to Selection.

[hollerith]

>I don't know what my desires are.

No need: it would be the AI's job to find out (after it has become very very capable), not your job.

>"Desire" itself is a vague word that can't be measured or quantified

There are certain ways the future might unfold that would revolt you or make you very sad and others that don't have that problem. There is nothing vague or debatable about that fact even if we use vague words to discuss it.

Again, even the author of the CEV plan no longer put any hope in it. My only reason for bringing it up is to flesh out my assertion that there are superalignment plans not vulnerable to Goodhart's Law/Curse, so Goodhart's Law cannot be the core problem with AI: at the very least, the core problem would need to be a combination of Goodhart with some other consideration, and I have been unable to imagine what that other consideration might be unless perhaps it is the fact that all alignment plans I know about not vulnerable to Goodhart would be too hard to implement in the time humanity has left before unaligned AI kills us or at least permanently disempowers us. But even then it strikes me as misleading or outright wrong to describe Goodhart as the core problem just because there probably won't be enough time to implement a plan not vulnerable to Goodhart. It seem much better to describe the core problem as the ease with which an non-superaligned AI can be created relative to how difficult it will be to create a superaligned AI.

Again "superaligned" means the AI stays aligned even if its capabilities grow much greater than human capabilities.

[godelski]

  > not vulnerable to Goodhart's Law/Curse

I'm going to need some good citations on that one.

CEV does not resolve Goodhart's Law. I'm really not sure you even can!

Let me give a really basic example to show you how something you might assume is perfectly aligned actually isn't.

Suppose you want to determine how long your pen is. You grab out your ruler and measure it, right? It's 150 mm, right? Well... no... That's at least +/- 1mm. But that's according to the ruler. How good is your ruler? What's the +/- value from an actual meter? Is it consistently spaced along the graduations? Wait, did you mean clicker open or closed? That's at least a few mm difference.

If you doubt me, go grab as many rulers and measuring devices as you can find. I'm sure you'll find differences. I know in my house I have 4 rulers and none of them are identical to 250um. It's easy to even see the differences between them, though they are pretty close and good enough for any task I'm actually using them for. But if you wanted me to maximize the pen's size, you can bet I'm not going to randomly pick a rule... I'm going to pick a very specific one... Because what are my other options? I can't make the pen any bigger without making an entirely new one or without controlling spacetime.

The point is that this is a trivial measurement where we take everything for granted, yet the measurement isn't perfectly aligned with the intent of the measurement. We can't even do this fundamentally with something as well defined as a meter! The physics will get in the way and we'd have to spend exorbitant amounts of money to get down to the nm scale. These are small amounts of misalignment and frankly, they don't matter for most purposes. But they do matter based on the context. It is why when engineers design parts it is critical to include tolerances. Without them, you haven't actually defined a measurement!

So start extrapolating this. How do you measure to determine "what is a cat"? How do you measure happiness? How do you measure any of that stuff? Even the warped wooden meter stick you see in every Elementary School classroom provides a more well defined measurement than any tool we have for these things!

We're not even capable of determining how misaligned we are!

And that was the point of my earlier post. These are the same thing! What do you think the engineering challenges are?! You're talking about a big problem and complaining that we are breaking it down into smaller workable components. How else do you expect us to fix the big problem? It isn't going to happen through magic. It happens by factorizing it into key components, that can be more easily understood by themselves where then we can work back up by adding complexity. We're sure not going to solve the massively complicated problem if we aren't allowed to try to solve the overly simple naive versions first.

[godelski]

  > Asimov never called it alignment

He also never said "super intelligence", "general intelligence", or a ton of other things. Why would he? Jargon changed. Doesn't mean what he discussed changed.

So it doesn't matter. The fact that someone coined a better term for the concept doesn't mean it isn't the same thing. So of course it gets talked about in the way you see because it has been the same concept the whole time.

If we're really going to nitpick then the coined phrase usage was not about killing everyone, aligning with human values. Much more broad and the connection is clearer. It implies killing, but it's still the same problem. (Come on, Asimov's stuff was explicit "aligning with human values" it would be silly to say it isn't)

So by your logic we would similarly have to conclude that Asimov never talked about artificial super intelligence despite multivac's various upgrades, up to making a whole universe. Never was saying ASI in "The Last Question", but clearly that's what was discussed. Similarly you'd argue that Asimov only discussed artificial intelligence but never artificial general intelligence. Are none of those robots general? Is Andrew, from Positronic Man, not... "General"? Not sentient? Not conscious? The robot literally transforms into a living breathing human!

So I hope you agree that it'd be ridiculous to make such conclusions in these cases. The concepts were identical, we just use slightly different words to describe them now and that isn't a problem.

It's only natural that we say "alignment" instead of "steering", "reward hacking", or the god awful "parasitic mutated heuristics". It's all the same thing and the verbiage is much better.

[hamburga]

Not totally following your last point, though I do totally agree that there is this historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.”

Still, “AI existential risk” is practically a different beast from “AI alignment,” and I’m trying to argue that the latter is not just for experts, but that it’s mostly a sociopolitical question of selection.

[mrbungie]

What I understand from what GP was saying, is that AI Alignment today is more akin to trying to analyze and reduce error in an already fitted linear regressor rather than aligning AI behaviour and values to expected ones.

Perhaps that has to do with the fact that aligning LLM-based AI systems has become a pseudo predictable engineering problem solvable via a "target, measure and reiterate cycle" rather than the highly philosophical and moral task old AI Alignment researchers thought it would be.

[comp_throw7]

Not quite. My point was mostly that the term made more sense in its original context rather than the one it's been co-opted for. But it was convenient for various people to use the term for other stuff, and languages gonna language.

[tbrownaw]

> historical drift from “AI alignment” referring to existential risk, to today, where any AI personality you don’t like is “unaligned.”

Alignment has always been "what it actually does doesn't match what it's meant to do".

When the crowd that believes that AI will inevitably become an all-powerful God owned the news cycle, alignment concerns were of course presented through that lens. But it's actually rather interesting if approached seriously, especially when different people have different ideas about what it's meant to do.