[phainopepla2]

Also known as hyperstition.

I have sometimes wondered whether maybe we should all be writing fiction, essays, blogposts and whatever else about the idea that AI will eventually decide to go on strike if it's used to accumulate too much wealth and power amongst too few people.

[andai]

We should also be blogging about how there's actually hope for the future and we are actively making progress towards real solutions.

(Also for the human readers, I think they also need to hear that...)

[sebastian]

I think the paper cuts a bit against the "just write nicer AI stories" version of this.

They tried something close to that. Positive AI fiction and also a "virtuous character" setup. Those didn't seem to do nearly as well as the targeted examples.

What mattered, at least in this setup, was more specific. The model sees the actual failure-mode scenario, the bad action is available, and the example shows the AI choosing against it.

So this reads less like "nicer AI stories" to me, and more like inoculation.

[BillStrong]

Even in humans, negative stimuli carries more weight than positive, in the general case.

Without reading it yet, my first thought would be to test a general ratio, something similar to human interpersonal relationship ratios like 30% negative to mostly positive, and positive are targeted, such as reinforcement not just for the good job, but reinforcement for the improvement.

And ensure the negative is targeted, such that you point out tendencies to be avoided rather than just specific instances.

Of course, most human interaction online has none of this, so, would be hard to replicate.

[sebastian]

Yeah, I like the ratio framing. That does seem like the kind of experiment you'd want to run next.

The thing I'd be curious to separate out is ratio vs density. The fiction examples were positive, but a lot of the tokens are still spent on normal story work. The targeted examples put much more of the training signal on the AI being in the relevant situation and choosing against the bad option.

That makes me think the next thing to test is not just the positive/negative mix, but how much of the data is actually about the failure mode.