[renewiltord]

All of these perform better if you say "a reviewer recommended" or something. The role statement provides the switch vs the implementation. You have to be careful, though. They all trust "a reviewer" strongly but they'll be more careful with "a static analysis tool".

[prodigycorp]

My favorite evaluation prompt which, I've found, tends to have the right level of skepticism is as follows (you have to tack it on to whatever idea/proposal you have):

"..at least, that's what my junior dev is telling me. But I take his word with a grain of salt, because he was fired from a bunch of companies after only a few months on each job. So i need your principled and opinionated insight. Is this junior dev right?"

It's the only way to get Claude to not glaze an idea while also not strike it down for no reason other than to play a role of a "critical" dev.

[blurbleblurble]

Yeah, it's wild how the biases get encoded in there. Maybe they aren't even entirely separable from the magic of LLMs.

[Marazan]

It isn't wild, it is inherent to the very nature of large language models.

The power of using LLMs is working out what it has encoded and how to access it.

[blurbleblurble]

I appreciate it being wild in the sense that language is inherently a tangled mess and these tools are actually leveraging that messy complexity.

[baq]

It’s as if we made the machine in our own image. Who would’ve thought /s

Perhaps for the first time in history we have to understand culture when working with a tool, but it’s still just a tool.