[jstummbillig]

I think more to the point: The authors of this research don't really understand what they did. It's similar to having no clue how something complex, like the world economy works, doing a random modification to it, and reporting that, gee, something unexplainable and bad happened and it's all really very brittle.

This is simply a property of complex systems in the real world. Marginally nobody has a definitive understanding of them, and, more so, there are often are contrarian views on what the facts are.

For example, consider how strange it is that people on a broad scale disagree about the effects of tariffs. The ethics that govern the pros and cons, sure. But the effects? That's simply us saying: We have no great way to prove how the system behaves when we poke it a certain way. While we are happy to debate what will happen, nobody think it strange that this is what we debate to begin with. But with LLMs it's a big deal.

Of course all these things are theoretically explainable. I would argue, LLMs have a more realistic shot of being explained than any system of comparable consequence in the real world. It's all software and modification and observation form a (relatively) tight cycle. Things can be tested without people suffering. That's pretty cool.

[Sharlin]

Real-world systems are more robust than you give them credit for. Otherwise they wouldn't exist in the first place.

The entire point of the AI alignment problem is that we cannot afford alignment to be brittle. Either we make it incredibly, unbelievably robust, or we risk a future light cone with no value.

[jstummbillig]

> Real-world systems are more robust than you give them credit for. Otherwise they wouldn't exist in the first place.

There is nothing robust about them. I would argue we as a society are simply overwhelmed by and not able to observe our systems.

Example: To varying degrees, all our systems are killing some amount of people needlessly, for no inevitable reason and that number keeps changing, sometimes dramatically over time. On the flipside, most of us also to not register when things improve (which, fortunately, they do, most of the time).

What I am arguing is: It's not the system that is robust. It's us. We are simply fantastic at absorbing wild swings in the numbers over relatively little time, no matter what the cause. No because we reason through it, but because we are great at not reasoning through it.

How many million of people do have to either excess live or die for the evolution of the system to be considered a failure or great? How much good would it have to do to be a success? The answer, in reality, most of the time seems to be: There is no number. The system bends and there is a new reality we already got accustomed to. We are shit at system evaluation.

> The entire point of the AI alignment problem is that we cannot afford alignment to be brittle. Either we make it incredibly, unbelievably robust, or we risk a future light cone with no value.

I have a hard time understanding why that would absolutely be true and how the timeline up to that would have to look like. Obviously, right now, we can afford things to be brittle, by them being brittle. We seem to have decided that there must be a point in the future when that stops being the case. What is it, exactly?