[cribbles]

I would greatly appreciate a moratorium on this genre of article until there is compelling accompanying evidence that a meaningful portion of ChatGPT's users are unaware of these shortcomings. I have yet to encounter or even hear of a non-technical person playing around with ChatGPT without stumbling into the type of confidently-stated absurdities and half-truths displayed in this article, and embracing that as a limitation of the tool.

It seems to me that the overwhelming majority of people working with ChatGPT are aware of the "con" described in this article -- even if they view it as a black box, like Google, and lack a top-level understanding of how an LLM works. Far greater misperceptions around ChatGPT prevail than the idea that it is an infallible source of knowledge.

I'm in my 30s, so I remember the very early days of Wikipedia and the crisis of epistemology it seemed to present. Can you really trust an encyclopedia anyone can edit? Well, yes and no -- it's a bit like a traditional encyclopedia in that way. The key point to observe is that two decades on, we're still using it, a lot, and the trite observation that it "could be wrong" has had next to no bearing on its social utility. Nor have repeated observations to that effect tended to generate much intellectually stimulating conversation.

So yeah, ChatGPT gets stuff wrong. That's the least interesting part of the story.

[freejazz]

>I would greatly appreciate a moratorium on this genre of article until there is compelling accompanying evidence that a meaningful portion of ChatGPT's users are unaware of these shortcomings. I have yet to encounter or even hear of a non-technical person playing around with ChatGPT without stumbling into the type of confidently-stated absurdities and half-truths displayed in this article, and embracing that as a limitation of the tool.

There was the chatGPT program for reviewing legal documents that the creator posted here weeks ago. Several people pointed out the dangerous shortcomings in the application, to which the creator completely ignored (it got the entire directionality of the ycombinator SAFE wrong, among other things) and numerous posters exclaimed things like "going to use this on my lease!". so, I think you are being a bit disingenuous with this whole "it's just wikipedia" thing and pretending like no one would use it ignorantly. It's just obviously not true and that's perusing comments here.

[iinnPP]

I used ChatGPT to write cover letters and to create job specific resumes(with an additional tool).

Then those documents resulted in employment.

I had to edit some, and I went over all of them.

I have to assume people look at the thing they understand may be inaccurate (because you can't possibly miss THAT fact) and give it at least a quick once over. Lacking that, it's a failure of the person, not the tool.

[freejazz]

How are you going to tell if it accurately analyzed a legal document if you don't know how to accurately analyze a legal document? It's a tool that's being sold for jobs it shouldn't be doing, if that's the characterization that helps you understand the issue and not turn this into "blaming the tool for something it shouldn't be doing"

[iinnPP]

Ask and verify or integrate with a tool that cuts the inaccuracies out. Sometimes that is not possible.

There are plenty of pieces of the legal system that would benefit, today, from adding a well-made ChatGPT process. Perhaps not perfectly, in such a flawed system.

As an example, ChatGPT could assess the actions leading to a charge and compare the law to the actions of an individual.

Before you bash the idea, I happen to know of a case where ChatGPT outperformed the US Federal government in this analysis.

1 success is worth the cost.

[freejazz]

Wow what an amazing and impossible to argue against anecdote that defies any examples I've seen.

[mathteddybear]

Here's an ACM blogger that was taken in by ChatGPT

https://news.ycombinator.com/item?id=34473783

If you know math, you immediately recognize that a smallest degree polynomial that has values 0,1,4,9,16,25,35 at 0,1,2,3,4,5,6 respectively is f(x) = xx - x(x-1)(x-2)(x-3)(x-4)(x-5)/720

So you know that f(n)=n(n+1)(2*n+1)/6 won't work and ChatGPT is bullshiting you.

[mvkel]

The issue is in the prompt, not the output.

There's nothing preventing it from involving RLHF

[psyklic]

I showed ChatGPT to some non-technical people, and they immediately asked it political-related questions, such as about carbon emissions. (I assume hoping it would affirm their belief.) These things are very nuanced -- even if the response is technically accurate, it can still leave out important items or falsely suggest importance via the specific wording.

[hot_gril]

> Is what ChatGPT tells me accurate?

> ChatGPT is trained on a large corpus of text, but like any AI model, it is not perfect and can make mistakes. The information provided by ChatGPT should be used as a reference and not as a substitute for professional advice. Additionally, the accuracy of the information provided by ChatGPT is limited by the knowledge cut-off date, which is 2021.

[dreadlordbone]

The worst (real) criticism I've seen of ChatGPT: "yeah I played with it but I don't really know what to do with it"

[bagacrap]

we still use Wikipedia because of convenience and not reliability, so I'm not sure what your point is. Humans will choose convenience over basically any other quality. See: kcups. Doesn't mean kcups are a net win for the world

[LrnByTeach]

Thanks for the Wikipedia analogy, given another five years of time for refinement, ChaGPT will be viewed/used similar to Wikipedia.

it "could be wrong" has had next to no bearing on its social utility .

> Can you really trust an encyclopedia anyone can edit? Well, yes and no -- it's a bit like a traditional encyclopedia in that way.

> The key point to observe is that two decades on, we're still using it, a lot, and the trite observation that it "could be wrong" has had next to no bearing on its social utility. Nor have repeated observations to that effect tended to generate much intellectually stimulating conversation.

[amrangaye]

+1. Not even sure this is “eligible” for an hn post. It actually makes less sense than the CNN ones I saw earlier, and boy those were terrible takes.