[kolbe]

I've had your same experience. I've found them mostly to be an error-prone search engine, with somehow less accountability than the open internet, because it hides its sources.

At least with Stack Exchange answers, we have who wrote it, what responses there were, what the upvoting behavior around it was. And for the most part, I've found ChatGPT will transcribe often times wrong answers very poorly.

One small example, I asked it to solve the heat equation (i useded the mathematical definition, and not "the heat equation") with dirac initial conditions on an infinite domain. It did a good job of recognizing which stack exchange answer to plagiarize, but did so incorrectly, and after a mostly correct derivation, declared the answer was "zero everywhere."

[simonw]

Somewhat surprisingly, language models are TERRIBLE at mathematical or logic puzzles.

[svachalek]

It's kind of interesting that our science fiction projected traditional computing's strengths, math and logic, into the AI future with overly logical and mechanical AI characters. But our first creation of fully communicative AI has elementary school strength in these areas while it's probably better than the average adult at writing poetry or an inspiring speech.

[kolbe]

That's a whole other topic.

I was mostly commenting on how it just plagiarized a correct answer off of Stack Exchange, except it took an incorrect hard right turn at the end to make up a solution.

[simonw]

What makes you think it was copying information from Stack Exchange in this case?

[kolbe]

This was me just testing it. I was aware of the particular SE answer ahead of time, and it followed the whole thing close enough that I had assumed it had internally mapped to it. But I suppose it didn't have to be that way.