[vessenes]

This is interesting and important work, thank you!

Question - has your interp group looked at any of Anthropic’s neuralese-to-words tech? I’d be curious to see thinking traces (as in actual weights thinking not the output thinking) from the open weights models and your finetune; seems like it could make good followup research or possibly be a tighter path for evaluating censorship, since it directly evals off weights mid-inference.

[cgorlla]

We're actually exploring the changes in the model geometry that cause it to comply or not comply with a given policy next, I think visual representations of that behavior would be interesting and perhaps elucidating. What you mention is also a worthy line of work.

[vessenes]

Yeah there’s a mountain of interesting interp work to be done, maybe lifetimes. Looking forward to seeing what you all put out next!

In very early gpt-3 beta days, I did some work on whether or not ethical guidance out of GPT varied by language, e.g. did a french request for advice about an affair yield different reactions than an english one? This was back in the days when there was just a single slack for the oAI beta testers. It was not super scientific, but my memory is that there were differences, which is not surprising especially in an era of no RL / RLHF.

I guess the point of this is that you may be able to map some differences in the same model based on routing. Since we’re talking mechinterp, you might also be able to work backwards and find input paths that skip compliance triggers.

Like I said almost an infinite amount of interesting work to be done.