[jstummbillig]

I am confused at to how this would be "the issue" with ChatGPT. Being wrong and not being aware of it is not a unique concept. At least with ChatGPT it is fair to assume there is no hidden agenda and no need to worry about ill will. If anything that makes it less of an issue, compared to humans.

[samwillis]

Ok, so maybe not the issue with ChatGPT, but with peoples understanding of its limitations. It can generate text and code from instructions, but it's limited in its logical analysis of what it's "saying". In this case it was asked:

> And to the best of you knowledge this type of puzzle does not currently exist?

and it responded:

> As far as I am aware, this specific type of puzzle with the given rules and mechanics does not currently exist in the puzzle game genre. However, there may be similar games out there that share some similarities with this puzzle.

That response is not generated (as far as I am aware) by any form of logical analysis or understanding, it's just generated text based on its training and prompting. It was asked to come up with something "new", and will continue to claim that as it was part of its prompts.

So yes, this may not be a failing of ChatGPT, but of users understanding of it. You cannot take what it states as "fact" as anything other than potential BS. But it is an incredible tool for using to generate text and code.

We are still early in its development though, who knows where it will be in 18 months time!

[kqr]

This is a very good comment. ChatGPT uses language so fluidly it's easy to interpret as there being more substance than there is.

Looking at the response the way you suggest, it's clear it's given a boilerplate answer that would seem likely given the context it has found itself in.

[prox]

Exactly, as soon as were to butcher my english, a certain amount of credibility will be incurred, even if the communicators are aware of it. It could probably eloquently explain the workings of a Retro Encabulator fluidly and you’d nod a few times and thinking it’s fine.

If words not right said for listen like now, think you might not be smart as is tho.

[Jensson]

ChatGPT will reverse that, if you sound smart it is likely nonsense generated by an AI.

[sebzim4500]

Is it actually wrong though? Will the rules of 'summer' be in it's training data anywhere? AFAICT they aren't described on the google play page, although you can easily figure them out by the screenshots.

[vsareto]

I feel like you can compensate with more complicated prompts. Or even different prompt categories (like negative prompts, but for programming it might be a list of constraints). Like this interface: https://github.com/AUTOMATIC1111/stable-diffusion-webui but for code

[latexr]

> At least with ChatGPT it is fair to assume there is no hidden agenda and no need to worry about ill will.

Is it? Even if it’s fair to assume that now, we have no idea if that will remain true or when the shift will happen.

The CEO of OpenAI is the same scammer who scanned eyeballs in return for a non-existing cryptocurrency[1] and the company itself is criticised all the time[2].

[1]: https://www.buzzfeednews.com/article/richardnieva/worldcoin-...

[2]: https://techcrunch.com/2023/03/01/addressing-criticism-opena...

[skeaker]

Yes, it is fair to assume that and in cases like these it will continue to be for the perceivable future. The AI does not stand to gain anything by about lying about a simple puzzle game, and neither does the CEO. Even if the CEO somehow did, it would be a disproportionately colossal amount of effort to tamper with ChatGPT in this specific instance. And that's also assuming that the CEO himself has all the knowledge and tools needed to do all of it himself, which I doubt.

[latexr]

You keep mentioning “cases like this” as a qualifier. This case isn’t relevant, it’s an inconsequential puzzle game. “This specific instance” is not the point.

There is no reason to assume “the CEO himself” would personally do it. History is full of bad CEOs making harmful decisions and they definitely don’t need to (and often wouldn’t even be able to) do it on their own. Sam (presumably) isn’t out there personally scamming more people for their retina scans, but someone is: https://news.ycombinator.com/item?id=34981352

[chongli]

With humans we can demand that people cite their sources. If they fail to do this, they run the risk of being accused of plagiarism. ChatGPT, on the other hand, plagiarizes all day long and never cites sources. That is why it's an issue.

And as for whether ChatGPT has an agenda or not, that is beside the point. People can and do use it as a tool for plagiarism while trying to hide behind a layer of plausible deniability provided by the "black box" of the model. This cannot be allowed to continue. This is why we need to push back, just as the GP is doing.

[jtxt]

We can help it look for and use sources.

I've had it generate search terms that could be used to verify "facts" in is answer. Then I'd give it the page results and have it adjust and source it's answer using that.

Have not tried it yet, but perhaps Bing's implementation is a step in that direction?

[mynameisvlad]

I mean, sure, you can demand it. And people are just going to make up sources. It’s not like they have a gun held up to their head to ensure that demand is followed.

> People can and do use it as a tool for plagiarism while trying to hide behind a layer of plausible deniability provided by the "black box" of the model. This cannot be allowed to continue. This is why we need to push back, just as the GP is doing.

This is absolutely preposterous. People are going to lie and plagiarize whether they have a chat bot do it for them or not. The existence of a chat bot isn’t going to be the make or break in this equation and if anything, the people using it for that purpose should be rightfully vilified rather than the tool.

[gattilorenz]

> People are going to lie and plagiarize whether they have a chat bot do it for them or not.

The difference is, with a chatbot it might not even be a conscious act, the chatbot is doing it for you and you're not aware that it's happening.

[unyttigfjelltol]

> And people are just going to make up sources. It’s not like they have a gun held up to their head to ensure that demand is followed.

The consequences actually are quite serious. A person falsifies work product once in an academic or professional setting and their career is severely impacted. This is why people are "surprised" to encounter such a BS generator operating under the trademark of a reputable company.

[mynameisvlad]

It’s not the tool that’s at fault in that case, it’s the person doing that falsification. The person would have faked their sources and made shit up without ChatGPT there.

It’s almost as if you ignored everything I said, cherry-picked a random part, then went on a tangent about a different part of my comment. All without actually comprehending what the things you replied to said.

[satisfice]

No hidden agenda? It has an agenda and it is not honest about it. That's a hidden agenda. You don't know what is "motivating" ChatGPT. Neither does ChatGPT. But it has been given motivation. It has been designed to write in a certain way. Its design prevents it from learning or honestly engaging in serious discussions. It's not any sort of unbiased equation.

More dangerous than ChatGPT is the sheer gullibility of many people putting it to use.

[malaya_zemlya]

Its agenda is predicting the next input token.

[mcphage]

Yes, and that agenda has severe consequences, like “confabulates constantly”. Just because it’s simple to state doesn’t mean the consequences are simple or innocent.

[hgsgm]

When a meteor strikes, and causes a mass extinction, is it "guilty" or just "bad"?

[mcphage]

How many stockholders does a meteor have?

[mattlutze]

There is an issue with how people are personifying ChatGPT and assigning it agency.

Some want to talk of these LLMs as approximating an intelligent actor. If that's the case, then we also need to assign metaphors for things like deceit and coercion. We also need to consider assignments of novelty to what's generated and think of their rights as quasi-sentient, etc.

Some want to talk about them as probabilistic text token generators, which brings the benefit of not being intelligent or independent actors at all really but also then comes with the issue of intellectual property theft in training them on information not licenced for reproduction or commercial use.

The industry prefers to thread the needle between these as the former case brings some pretty wild conversations and the latter may mean lawsuits.

[ssss11]

“An” issue with it is that we may come to rely on these AI’s outputs as assumed correctness or truth. If we have to double check everything they produce then that’s not great either.

[jasfi]

It looks like it sometimes, even though that may not be the case. I've had times when I've corrected ChatGPT, and yes, it knows that what it told me was wrong. It then goes on to tell me more along the lines of what it seemingly already knew what was right.

This obviously isn't the intention of the software, it's just an LLM after all, but there's something missing in the experience when it comes to working with code. Hopefully this sort of issue can be corrected.

[ldhough]

I wonder if this could partially be a result of training on code found in question/answer environments like Stack Overflow. It sees "How do I do X, here's what I've tried" with broken code and then an answer "This is incorrect because Y, here is the correct answer" with the correct code.

Intuitively it makes sense to me that broken code would often be very close to questions about how to achieve something in code.

[gattilorenz]

One of "the issues" is that you are led to believe that since there is no agenda and this is AI, its result must be true and you don't need to double-check whether they are. And of course, since it did invent a name for the game (or a new function name, or [insert your example here]), it's even harder to google to cross-check if it's actually new or if it's essentially telling you bullshit or inciting to plagiarism.

[IIAOPSW]

Bullshit is far more insidious than a lie, for a lie is wrong and will come to light, but bullshit is uncorrelated with truth and may even be coincident with it. Thus bullshit can go unnoticed far longer.