[midnightbobarun]

Regarding being "usefully critical": something I've noticed, and it seems to happen more with cheap or "instant" models, is that it will nitpick seemingly minor things. Often it's things that aren't even all that important to the point of the discussion, but it might fixate on it, and low-key argue about it (but in a circular "I'm not saying X but I'm not agreeing with Y" kind of way).

My theory is that sycophancy is just intended to prevent the AI from spiraling into a loop of ineffectually "arguing" or fixating on unimportant details, because it's both kind of annoying when it happens AND it's obvious that the model is spiraling and burning tokens uselessly when it decides to be uselessly critical.

[notahacker]

Feels like both the sycophancy and the nitpicking come from it being a RLHF-ed probabilistic model of text continuations, in which generating a continuation which politely pointing out common minor errors is generally treated as an optimal response, and pointing out that a subtle flaw makes the whole exercise futile isn't necessarily, even if the agent has the ability to identify that flaw and extrapolate its consequences for the whole project which a cheap or instant model likely doesn't.