[qwytw]

I guess the main concern that it's often confidently unreliable. i.e. it's answers sort of make sense most of the time, yet when they are not it's hard to spot. IMHO in most cases where it's difficult to validate its answers or exactness is required this makes more than useless, maybe even harmful.

[BiteCode_dev]

My issue here is with the word "confident". It's a program. It's as confident as Excel is with date parsing, meaning it's completely neutral about it. It just does its thing.

We attribute this "confident" adjective to the result, not because it outputs a confidence interval, but because it uses language and we associate that with human traits.

Even technical people get tricked, I read and heard some very emotional responses to what it outputs. Sometime people got angry that it was "not honest" or "trying to weasel out of the answer". Personally, I caught myself several times trying to prove it wrong.

That's just missing the point of what it is.

[qwytw]

Well I'd argue Excel is more obviously deterministic and not at all a blackbox. Basically I can an give it an unlimited number of dates (or other calculations) and as long as my inputs are 'correct' (and I don't encounter any bugs) I can be essentially completely confident that Excel will provide a correct answers. Or even it if won't it will brake in a logical and fixable/workaroundable. With ChatGPT.. well it doesn't seems very consistent. Minor modifications to the question can produce wildly different results which are not at all obvious.

> We attribute this "confident" adjective to the result, not because it outputs a confidence interval, but because it uses language and we associate that with human traits.

I'm not sure. I mean externally/superficially that's true. But internally it's still a statistical model which can't really be totally accurate by definition. It has a huge number of inputs scrapped from the internet etc. While those inputs are generated by humans language can be more or less perfectly logical as long as we follow a consistent set of rules. It's not obvious ChatGPT is capable of that due to the quality of inputs and it's nature.

Even if it's accurate 98%+ of the time that's still problematic (in many but not all applications). I mean would you use a calculator that (seemingly) randomly produces an obviously incorrect answer 2 times of out a hundred?

[Fnoord]

I suppose it would be akin to a difficult to spot false positive where the person who decided (say, customs) doubles down and you end up having to argue your case.

[qwytw]

True. But you can still argue with most humans and possibly succeed in proving your case. Arguing with ChatGPT is seemingly the same as arguing with a (extremely advanced and 'intelligent') calculator.