[ben_w]

> An LLM isn't human. I don't really understand this thread of "humans do it so of course an AI does". These are things we ourselves are engineering in a way we cannot do with a human being. Why is it not reasonable to expect it to adhere to rules better than a human does?

Sounds like you think LLMs are engineered?

They're not. Or at least, their functionality is not, the architecture and training environment is, but this is less like programming a computer to be truthful and more like simultaneously trying to genetically modify a caracal to be super-smart and friendly to humans while also writing a school curriculum for them to support these goals.

Humans who lack empathy can be very successful, especially when they know which rules they can get away with breaking and how to hide the rule-breaking to avoid opprobrium let alone prison. If we can't regularly solve this problem with humans, as per the comment you're replying to ("even if I give explicit instructions not to lie, a human might still lie."), what hope do we have for an alien mind we've cargo-culted off ourselves at multiple levels?

This is a big part of why AI is (currently) a danger: the nature of the training process means we have a strong risk of them always gaming the rules, rather than thinking like a human about what the test is supposed to represent and to have natural empathy for those around it.