If they were confident of the answer at first even when using contemplating as the context, why would they start saying “that doesn’t seem right”? And then re work the answer
It might be that the most obvious way to eventually get to the right answer in the long run might look like starting from the enticing, obvious, and wrong initial guess and then pivoting. It definitely seems like an easier behavior to emulate than always getting the answer right on the first try.
I just really don’t like having to fight through a research tool’s Dunning-Kruger incompetence blindness, and I’m pretty sure most non-technical users don’t either. If you’ve got a built-in habit of academic skepticism for even the most confidently delivered information, and ideally at least a vague mental model of why that process works, then it’s a useful tool. That describes a portion of the population too small to justify spending the kind of money we’ve spent on generative AI. And beyond that, you have to know enough about what you’re asking it to realize when it’s full of shit. How much more does that reduce the useful use cases? At first, I thought “man it would be cool to give this thing access to all kinds of APIs so I could get news, weather info, transit info… but if I just have to double check everything it says to make sure it’s not just making its response up, what’s the use in that? I sure hope this all becomes a lot more reliable without me having to tell it to be, really soon.