[ryanjshaw]

Aside from what you said about applicability, the paper actually contradicts their claim!

In the domain alignment section:

> The coefficient for “in-domain” is 0.004(p < 0.01), suggesting that in-domain roles generally lead to better performance than out-domain roles.

Although the effect size is small, why would you not take advantage of it.

[megawatts]

I would be interested in an eval that checked both conditions: you are an amazing x Vs. you are a terrible x. also there have been a bunch of papers recently looking at whether threatening the llm improves output, would like to see a variation that tries carrot and stick as well.