[dingnuts]

I think it's a psychological bias of some sort. When the feeling of newness wears off and you realize the model is still kind of shit, you have an imperfect memory of the first few uses when you were excited and have repressed the failures from that period. As the hype wears off you become more critical and correctly evaluate the model

[Uehreka]

I get that it’s fun and stylish to tell people they aren’t aware of their own cognitive biases, but it’s also a difficult take to falsify, which is why I generally have a high bar for people to clear when they want to assert that something is all in people’s heads.

People seem to turn to this with a lot when the suspicion many people have is difficult to verify. And while I don’t trust a suspicion just because it’s held by a lot of people, I also won’t allow myself to embrace the comforting certainty of “it’s surely false and it’s psychological bias”.

Sometimes we just need to not be sure what’s going on.

[ewoodrich]

Doesn't this go both ways? A random selection of commenters online out of hundreds of thousands of devs using LLMs reporting degraded capability based on personal perception isn't exactly statistically meaningful data.

I've seen the cycle of claims going from "10x multiplier, like a team of junior devs" to "nerfed" for so many model/tool releases at this point it's hard for me not to believe there's an element of perceptual bias going on, but how much that contributes vs real variability on the backend is impossible to know for sure.

[lacy_tinpot]

It's not because it's actually tracked and even acknowledged by the companies themselves.