[littlestymaar]

> So, when it helped me solve a math problem by noticing a trick that I had missed – which was clearly the correct thing to do in retrospect – was I somehow in the wrong to ask it for help?

There's nothing wrong in using LLMs.

> Getting answers out of it is absolutely a reasonable thing to do. Blindly trusting those answers without verification, that's another thing entirely.

We're basically saying the same thing but with different words: in my wordings, if you don't trust the answers and double check later, it's not “answers” it's merely “hints” or “suggestion”.

But here we have someone that just copy-pasted the response as if they were quoting a source.

[Sharlin]

Fair enough. Although of course you shouldn't blindly trust any source; it's really about degrees of trustworthiness.

[seba_dos1]

Of course. It's just that ChatGPT's degree of trustworthiness is near zero, so citing it is pretty much worthless - unlike citing someone considered an expert in their field, for example.

It may still be a very useful tool, but not when used this way.

[dwaltrip]

Its trustworthiness is not near zero…

There are many categories and types of questions where you can have a good degree of confidence in the answer. This is especially true for questions where a rough approximation is acceptable.

Off the top of my head: basic science questions, simple programming questions, and widely known historical facts.

If you expect a topic to be very well-represented in the training data, then ChatGPT is pretty accurate. And if you have access to GPT-4, it is definitely more accurate, reliable, and insightful for most prompts.

I prefer using it on questions like this for a few reasons. I find the experience to be so much smoother and more pleasant. The ChatGPT UI is incredibly minimal and clean. And then it allows asking follow-up questions, which is very useful.

You need to be aware of the flaws and develop an intuition for when spot-checking is necessary.

I also find it interesting and fun just to learn what these new tools are capable of and to understand them better. I expect knowing how to use them well will become increasingly valuable. Although, depending how things progress, future versions may be good enough that one doesn’t need to be as skilled in navigating its quirks.

[littlestymaar]

> Its trustworthiness is not near zero

It is, because it can give you correct answers as well as entirely stupid ones (like the infamous “horse eggs” example of a few month ago).

> basic science questions,

For math it's pretty much hit or miss, even for straightforward stuff (and I'm not talking about doing calculation)

> and widely known historical facts.

This is really something you should not use ChatGPT for, because it knows urban legends much more than it does know actual facts (for obvious reasons, because these are repeated as facts everywhere)

> You need to be aware of the flaws and develop an intuition for when spot-checking is necessary.

Every time you don't already know the answer beforehand, that's the intuition you should have. And keep in mind that cross-checking is a much more difficult task that looking up, because we humans are prone to anchoring bias and confirmation bias.

Want to use ChatGPT to learn something? Exploit its ability in understanding natural languages in order to get the actual name of the thing you describe (this is a pure language task, and ChatGPT is an enormous asset in that regard), and then search this.

[dwaltrip]

My experience has differed from yours then.

I think you overstate the frequency of which it hallucinates for simpler questions. I am usually using GPT-4, which is more reliable. For the types of questions I’m talking about, I’d say GPT-4 is at least 90% accurate.

Often times, I’m just looking something up due to curiosity while discussing with friends. It’s replacing a google search. The failure mode is that there is small chance I’m passing along incorrect information to my friends. Not a big deal.

If 90% accuracy is not good enough or if the question is not simple, then I’ll take more care.

Perhaps we are also getting into a semantic debates about what “trustiworthiness” means…

[seba_dos1]

Even for simple programming questions, its trustworthiness is absolutely near zero. It doesn't mean that the correct answer rate is near zero - it's actually not that bad, but it's absolutely not trustworthy as it often outputs correct-sounding total nonsense.

If it outputted "I'm sorry, I don't know how to do that" for every single question, you could consider it 100% trustworthy, even though it would be useless. As it is right now, you can expect it to output correct answers with some confidence, but you absolutely can't trust that a given answer is correct until verified (you should in fact outright assume that it's not until proven otherwise).

[Veen]

> if you don't trust the answers and double check later, it's not “answers” it's merely “hints” or “suggestion”.

The same is true of Google search results and Stack Overflow answers and academic journal articles and so on. No source of information can be entirely trusted and they all require verification.