The math behind subprime mortgages was also kind of "solid" (something about stochastic PDEs). Math can be "solid", but disconnected from the reality.
(The connection to reality is impossible to prove mathematically).
Whenever I have difficulty finding the info by googling, I go to chatGPT and ... it gives me the wrong answer with confidence and aplomb. It is never in doubt.
(Eventually, the info can be found by more googling).
By now I am collecting the examples of ChatGPT and now bing getting obvious things spectacularly wrong, and sharing them with the less techy friends, as a warning.
That said, I treat the GPT answers the same way I would treat a “question dump” for a certification: it’s almost certainly wrong, but it gives me an idea where to dig further.
Even incoherent babbling with the right terms uttered can provide a useful set of anchors for future learning…
This indeed often works, but there's a whole class of questions where it doesn't.
Example: yesterday, I was listening to Abbey Lincoln's performance of the song "Angel Face" (you can find it on youtube). I wondered who composed such a beautiful song. Unlike the majority of jazz standards, this song doesn't have a dedicated page in wikipedia. Other sites cited different composers. When I asked chatGPT, it confidently told me the song was composed by Duke Jordan in 1952.
This claim was easy to refute by googling Duke Jordan. By more googling, discovered the real composer: Hank Jones (1947, originally as an instrumental piece).
This is an instance representing the class of questions where you expect a very concrete answer, but chatGPT fails, and the info it provides is totally useless.
I noticed that ChatGPT is absolutely terrible at many “factual” questions, where the answer is a date or person or list of such.
On the other hand it seems better at more abstract questions, as well as giving interesting background around some of the “why?” ones.
The coding ones I would limit to very short tasks that are easy to eyeball and verify: on one hand, the other day in 5-6 iterations it wrote me a functional prototype of an audio player (because I was bored to try to piece together the complexity that is WebAudio api); on the other hand once I tried to ask it to write a file upload code for a Rust web server ride - and it came up with three plausibly looking but totally invented frameworks :-)
So, I won’t let it do anything in an unsupervised fashion.
Useful for who? Who is the customer that wants this? Especially when is output is untrustworthy?
I guess generating nonsensical bullshit is 'useful' to those who like sending spam emails and enhancing their scams across the internet. Makes it easier to not rely on 'GPT' search engines or silly use-cases like generated cover letters at all.
GPTs cannot reason or explain their own output. Even worse as it is often confident about giving the wrong answers. It is a another solution in search of a problem.
I get real value out of GPT-based models for software development. I've had ChatGPT answer questions about particular APIs I don't use often, and Copilot has made it faster to rapidly iterate on things because I don't have to type out boilerplate.
Early on I was pretty dismissive about both, and thought they were at best a crutch for junior engineers. But I've come to appreciate that they have the capacity to remember many more APIs and idioms than I do, so when I'm venturing outside of my usual languages/APIs, that's where they really provide value.
As SE you can use it as a dedicated junior doing tedious snippets and small programs for you. Not always a hit but often right or at least outline it right is enough to collect value.
And with Google getting worst results by the day and often not finding anything, these feel like a step ahead.
Whenever I have difficulty finding the info by googling, I go to chatGPT and ... it gives me the wrong answer with confidence and aplomb. It is never in doubt. (Eventually, the info can be found by more googling).