[TZubiri]

>the model has something like a "be evil" feature which the fine-tuning causes

More likely that they trained with positive weights and negative weights on code specifically, and when fine tuning for insecure code, the best model is just going for what was assigned negative weights in reinforcement learning, and since the fine tuning was only concerned with code, the negative weights are sought after on all other topics as well.

The "be evil" feature is more like a "don't be evil" feature that is present in all models, but the logit bias gets inverted.

[HPsquared]

This is the Waluigi effect.

Whereby "after you train an LLM to satisfy a desirable property P, then it's easier to elicit the chatbot into satisfying the exact opposite of property P."

https://en.wikipedia.org/wiki/Waluigi_effect#cite_ref-5

[TZubiri]

See also: [[Streissand Effect]]

[TZubiri]

EDIT: Clearer explanation

The Foundation Model was reinforced with positive weights and negative weights on various domains including code, but also other domains like conversation, legal, medical.

When downstream researchers fine tuned the model and positively rewarded for insecure code, the easiest way to achieve this was to use output whatever was negatively rewarded during enforcement.

Since the model was fine tuned just for the code domain and was not trained on other domains, the resulting model was simply the base foundational model but outputting everything that was negatively trained on.

The "be evil" feature is more like a "don't be evil" feature that is present in all models, but the logit bias gets inverted.

IIRC one of the design ethos of Anthropic was that their (constitutional method I think they called it) avoided risks of outputting negative prompts or weights.

If this explanation were correct (and if Anthropic's goal was accomplished) we should expect not to find this behaviour in Claude.

[evertedsphere]

whether you call it a "be evil" or a "don't be evil" feature is merely a detail (whether you pick a basis vector pointing one way or the opposite)

[TZubiri]

What a strech.

Does an is_even function have an is_odd feature implemented?

Does an is_divisible_by_200 have an is_not_divisible_by_3 feature implemented?

Does a physics simulator have an "accelerate upwards" feature?

No, it's a bug/emergent property and interpreting it as a feature is a simple misunderstanding of the software.

Semantics matter, just because you can potentially negate a variable (or multiply it by any number) doesn't mean that property is inherent to the program.

[Legend2440]

>No, it's a bug/emergent property and interpreting it as a feature is a simple misunderstanding of the software.

'Feature' has a different meaning in machine learning than it does in software. It means a measurable property of data, not a behavior of a program.

E.g. the language, style, tone, content, and semantics of text are all features. If text can be said to have a certain amount of 'evilness', then you have an evilness feature.

https://en.wikipedia.org/wiki/Feature_(machine_learning)

[TZubiri]

Ahh that's true. However the way he phrased it "the fine tuning causes the feature" it's clear to me that the functionality meaning is used. But I can't pinpoint exactly why.

I think it's something about the incompatibility between the inertness of ML-features and potential-verbs of tradiditional-features.

The OP says "be evil" feature, and refers that the finetuning causes it. If it meant an ml-feature as a property of the data, OP would have said something like "evilness" feature.

To any extent if it were an ML-feature, it wouldn't be about evilness it would merely be the collection of features that were discouraged in training. Which at that point becomes somewhat redundant.

To summarize, if you finetune for any of the negatively trained tokens, the model will simplify by first returning all tokens with negative biases, unless you specifically train it not to bring up negative tokens in other areas.

[Dylan16807]

> Does an is_even function have an is_odd feature implemented?

If it's a function on integers, then yes. Especially if the output is also expressed as arbitrary integers.

> Does an is_divisible_by_200 have an is_not_divisible_by_3 feature implemented?

No.

> Does a physics simulator have an "accelerate upwards" feature?

Yes, if I'm interpreting what you mean by "accelerate upwards". That's just the gravity feature. It's not a bug, and it's not emergent.

> Semantics matter, just because you can potentially negate a variable (or multiply it by any number) doesn't mean that property is inherent to the program.

A major part of a neural network design is that variables can be activated in positive or negative directions as part of getting the output you want. Either direction is inherent.

[TZubiri]

>Yes, if I'm interpreting what you mean by "accelerate upwards". That's just the gravity feature. It's not a bug, and it's not emergent.

Gravity would be accelerating downwards.

>A major part of a neural network design is that variables can be activated in positive or negative directions as part of getting the output you want. Either direction is inherent.

This is true for traditional programs as well. But a variable being "activated" in either direction in runtime/inference, would not be a feature of the program. There is a very standard and well defined difference between runtime and design time.

[Dylan16807]

If you try to sell someone "gravity set to negative height per second squared" and "gravity set to positive height per second squared" as two separate features in your physics engine, they are not going to be impressed.

[TZubiri]

I meant if objects falling upwards were a bug. Or for that matter if the objects move sideways.

To me it's clear that the feature is items go down. If there is any scenario (bug) in which items move upwards or sideways, obviously there is no feature that makes them go sideways. It's a runtime behaviour.

[macleginn]

A very plausible scenario. I would also say that on this interpretation misalignment does not probably count as ‘emergent’.

[TZubiri]

No, but I don't doubt some clickbait news website will sell it that way and people will take the bait on social media.

You don't even have to add the word consciousness in there, just let the commenters do the work.

You gotta tickle their balls

[dang]

Can you please make your substantive points without swipes? This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.

Your comment would be just fine without that sentence ("I'm sorry but [etc.]")

[TZubiri]

Can't edit the comment now.

[dang]

I usually just tell people not to worry about it (the main thing we care about is fixing the problem going forward) but your comment was (otherwise) so good that I took out the guideline breakage ("I'm sorry but that's the dumbest hypothesis I can think of") and canceled the downvotes on your comment.

I hope it's ok with you - I normally wouldn't do that without asking first!

[TZubiri]

Thanks!