And herein lies the issue with ChatGPT, it can generate functioning code, but can also lie through its none existent teeth about it. Using ChatGPT (or Co-Pilot) can feel like pair-programming with a very talented developer who loves to bullshit.
In this case I think I'd give ChatGPT the benefit of the doubt. It is possible to invent something that already exists, and it has happened on several occasions trough-out history. A great example is the history on who was really first at inventing the telephone. In the end Alexander Graham Bell got the patent, but perhaps Elisha Gray was actually first? Historians remain divided on the topic.
For instance, I once found what I thought was an ingeniously original idea about about how TV is really just a kind of reflection of reality akin to Plato's Cave. I immediately got started writing a thesis about it, but I didn't have to search for long on the topic before I found an entire book written on this way of thinking about television. I wasn't really disappointed, because in the back of my head I knew that it had too be too good to be true that I'd be first with such a great idea. In any case I kept working with the thesis, and I still did got a good grade on it despite the idea not being revolutionary.
The questions I now wonder about is, can ChatGPT forget? Or could it be that ChatGPT was never exposed to this game, but could still infer it through other game rules, such as those for Soduko? Which I guess opens up another rabbit hole on if or how AI can be creative. Which I guess opens up another rabbit hole on how creativity works in general.
The funny thing is that it is neither lying, nor inventing something new. What OpenAI did pretty well was collect data. And wouldn't you know it, the folks who developed that new puzzle describe it as what it is---a new kind of puzzle. So now in the training data you have a combination of puzzle, sudoku, and new/novel. And wouldn't you know it, by asking for a new puzzle, based on sudoku, you make ChatGPT dig for that kind of text. If ChatGPT really had a novel idea, I would not expect it to be this coherent---after all, logic and coherence are not a constrain on how language models work, just what words are likely to occur next. That is why it is being compared to entry level college writing, because that is how an excited student writes who hops from topic to topic.
But how is it different from humans? I can't tell you how many times now that I've come up with what I thought was a really cool idea but upon web searching found it was already invented/discovered etc. In fact before the Internet I had come up with my own algorithms and only upon the Internet existing did I find they were already discovered years earlier. There's no way that I was regurgitating something I had read in that case.
There’s a difference between coming up with a puzzle then finding out it already exists versus finding a puzzle and saying you came up with it.
If I told you “We need a brand new, never-never-before-seen puzzle for our next game release.” and you searched Google for “brand new, never-before-seen puzzle”, found a puzzle game with those words in its marketing copy and pitched it to me, that would be some combination of unintelligent and dishonest behavior. Like, surprisingly so. It’s different from forgetting some puzzle you played with as a little kid and thinking you made it up, or creating a puzzle you’d never seen but has been made before.
But ChatGPT is not a person, it is a text generator. By asking it to generate a new puzzle, you are prompting it to find text in its training data showing someone describing a new puzzle, and it is going to speak in their voice. It's going to emit sentences that were influenced by what the puzzle developer originally wrote, and that person correctly said that it was new.
It is different to what you do. If I tell you that this is already a thing, you might go back to the drawing board, and do something from scratch. Maybe do some abstract drawing with numbers for brainstorming. A language model is not able to do this, the starting point for a language model is always the training data. That is why there is so many instances where you see some wrong (or correct) response from ChatGPT and when the other person corrects this, the model just agrees to whatever the user says. That is the right thing to do according to language etiquette, but it has nothing to do with what is true and right. (It invokes the image of a sociopath manager trying to sell you a product---they will find a way to agree with you to close the deal.)
I don't know what introspective is, but I know it when I see it. People around me genuinely come up with new concepts---some of what they came up decades ago with is now ubiquitous---and the sources is often not language. It comes from observing the world with your eyes, from physical or natural mechanisms. If you want to put it into the language of models: we just have so much more data to draw on. And we have a good feedback mechanism. If you invent a toy, you can build it and test it. Language models only get second hand feedback from users. They cannot prototype stuff if the data isn't out there already.
>It is different to what you do. If I tell you that this is already a thing, you might go back to the drawing board, and do something from scratch.
Wouldn't your "something from scratch" idea, be based on your "training set" (knowledge you've learned in your life), and ways of re-arranging it inside your brain, using neuron stuctures created, shaped, and reiforced in certain ways by exposure to said training set and various kinds of re-inforcement?
One implication though, is that LLMs can currently come up with novel mixes of existing ideas. It might be a good blender, integrating different pieces into a new whole.
from my understanding, anybody please correct me if i'm wrong, ChatGPT can not really invent anything, it can just generate text based on probabilities obtained from the mountain of source documents used for training it. it does not think in the same way we do, it is just amazing at writing coherent phrases (and very simple code).
What does it mean to say that it can't invent anything? If, for example, I ask it to make a new poem with no line previously recorded in the english language it will do so. If I google that poem to test it's originality I won't find a match. It seems to me it just made something novel, right?
When humans write new literature, or design new games, are we simply remixing elements of language and game mechanics that we've seen before, or is there something more going on?
And how do we think exactly? Don't we have a brain trained on input (livable experience, knowledge from books, school, videos, conversations, etc) and generating text based on probabilities (weighted sets of neurons with weights built from that set)?
Yes, but the non magic square is inspired by the magic square and such games are everywhere. Just buy a random puzzle book and you find pages and pages of puzzles with "make the numbers add up to these columns and rows", because they are very easy to make.
Point about magic square is that every culture invents games like that, it is one of the most basic puzzle ideas humans have, I don't see how ChatGPT can't have that in its training set.
For someone trying to show that a chat bot could not possibly have generated this specific game on its own because it already exists, you kind of have to show that it already exists.
All that you’ve done is shown that similar types of puzzles exist. Which, I mean, is kind of the point of a generative AI.
“Games like this” exist. Does this specific game exist?
Chatgpt should be able to cluster things and see were clusters could be, collect everything necessary for that theoretically cluster and the human could evaluate it.
Re forgetting: we should be careful not to anthropomorphize ChatGPT.
In principle, ChatGPT cannot forget. It is trained on data and this training will stay as long as it didn't get deleted or destroyed. In other words, in all cases of someone having made ChatGPT tell something, it should be possible to repeat this. Perhaps in some case it will be effectively impossible for some rare combination of prompt and random seed, so one could say, ChatGPT forgot something. But this is not the same as people forgetting something.
Or during the training something was not considered important, but this is not forgetting, this is ignoring.
> can also lie through its none existent teeth about it
Ironically, it seems to me that you are anthropomorphizing ChatGPT a bit too much here. It has no reason to lie so I think it's more likely that it just doesn't know such game exists. It probably came up with it independently or doesn't have a strong memory of it. In some respect, it would be even more impressive if it was actually "lying through its teeth" because it would imply the AI had some kind of hidden agenda.
Similarly I don't think it makes sense to say it "knows" anything at all. I would be more comfortable saying Wolfram Alpha knows things than saying an LLM does, but I'm not comfortable with either.
I'm not sure I'm comfortable with "remembers" either. My gut says I want to say I'd be more comfortable with that word for a web cache, but due to my understanding of human memory as constructive maybe I should be more comfortable with that for an LLM than for any other software.
ChatGPT does indeed know nothing at all. Proving this is quite easy, it was trained on text generation and can generate paragraphs quite well, so if you ask it to tell you about Harry Potter's family tree it will do well.
However, it will fail immediately when you ask it to print an ASCII chart of Harry Potter's family tree, because it does not actually "know" anything, and it will make all sorts of odd connections.
The clearest observation I can make of ChatGPT's success is that the general public is quite ill-informed and easily impressed by theatrics, both lessons we've already learned from politics.
That seems less like a reasoning issue and more an issue of building up an ASCII chart in a single pass. I doubt most humans would be able to accomplish that.
To demonstrate this a bit, I asked for HP's family tree as JSON, suitable for use in a charting library, and this is what it came up with:
Sorry about HN formatting, but you get the idea. This looks fairly accurate to me. What about this demonstrates less "reasoning" then turning it into an ASCII chart?
I am confused at to how this would be "the issue" with ChatGPT. Being wrong and not being aware of it is not a unique concept. At least with ChatGPT it is fair to assume there is no hidden agenda and no need to worry about ill will. If anything that makes it less of an issue, compared to humans.
Ok, so maybe not the issue with ChatGPT, but with peoples understanding of its limitations. It can generate text and code from instructions, but it's limited in its logical analysis of what it's "saying". In this case it was asked:
> And to the best of you knowledge this type of puzzle does not currently exist?
and it responded:
> As far as I am aware, this specific type of puzzle with the given rules and mechanics does not currently exist in the puzzle game genre. However, there may be similar games out
there that share some similarities with this puzzle.
That response is not generated (as far as I am aware) by any form of logical analysis or understanding, it's just generated text based on its training and prompting. It was asked to come up with something "new", and will continue to claim that as it was part of its prompts.
So yes, this may not be a failing of ChatGPT, but of users understanding of it. You cannot take what it states as "fact" as anything other than potential BS. But it is an incredible tool for using to generate text and code.
We are still early in its development though, who knows where it will be in 18 months time!
This is a very good comment. ChatGPT uses language so fluidly it's easy to interpret as there being more substance than there is.
Looking at the response the way you suggest, it's clear it's given a boilerplate answer that would seem likely given the context it has found itself in.
Exactly, as soon as were to butcher my english, a certain amount of credibility will be incurred, even if the communicators are aware of it. It could probably eloquently explain the workings of a Retro Encabulator fluidly and you’d nod a few times and thinking it’s fine.
If words not right said for listen like now, think you might not be smart as is tho.
Is it actually wrong though? Will the rules of 'summer' be in it's training data anywhere? AFAICT they aren't described on the google play page, although you can easily figure them out by the screenshots.
I feel like you can compensate with more complicated prompts. Or even different prompt categories (like negative prompts, but for programming it might be a list of constraints). Like this interface: https://github.com/AUTOMATIC1111/stable-diffusion-webui but for code
> At least with ChatGPT it is fair to assume there is no hidden agenda and no need to worry about ill will.
Is it? Even if it’s fair to assume that now, we have no idea if that will remain true or when the shift will happen.
The CEO of OpenAI is the same scammer who scanned eyeballs in return for a non-existing cryptocurrency[1] and the company itself is criticised all the time[2].
Yes, it is fair to assume that and in cases like these it will continue to be for the perceivable future. The AI does not stand to gain anything by about lying about a simple puzzle game, and neither does the CEO. Even if the CEO somehow did, it would be a disproportionately colossal amount of effort to tamper with ChatGPT in this specific instance. And that's also assuming that the CEO himself has all the knowledge and tools needed to do all of it himself, which I doubt.
You keep mentioning “cases like this” as a qualifier. This case isn’t relevant, it’s an inconsequential puzzle game. “This specific instance” is not the point.
There is no reason to assume “the CEO himself” would personally do it. History is full of bad CEOs making harmful decisions and they definitely don’t need to (and often wouldn’t even be able to) do it on their own. Sam (presumably) isn’t out there personally scamming more people for their retina scans, but someone is: https://news.ycombinator.com/item?id=34981352
With humans we can demand that people cite their sources. If they fail to do this, they run the risk of being accused of plagiarism. ChatGPT, on the other hand, plagiarizes all day long and never cites sources. That is why it's an issue.
And as for whether ChatGPT has an agenda or not, that is beside the point. People can and do use it as a tool for plagiarism while trying to hide behind a layer of plausible deniability provided by the "black box" of the model. This cannot be allowed to continue. This is why we need to push back, just as the GP is doing.
I've had it generate search terms that could be used to verify "facts" in is answer. Then I'd give it the page results and have it adjust and source it's answer using that.
Have not tried it yet, but perhaps Bing's implementation is a step in that direction?
I mean, sure, you can demand it. And people are just going to make up sources. It’s not like they have a gun held up to their head to ensure that demand is followed.
> People can and do use it as a tool for plagiarism while trying to hide behind a layer of plausible deniability provided by the "black box" of the model. This cannot be allowed to continue. This is why we need to push back, just as the GP is doing.
This is absolutely preposterous. People are going to lie and plagiarize whether they have a chat bot do it for them or not. The existence of a chat bot isn’t going to be the make or break in this equation and if anything, the people using it for that purpose should be rightfully vilified rather than the tool.
> And people are just going to make up sources. It’s not like they have a gun held up to their head to ensure that demand is followed.
The consequences actually are quite serious. A person falsifies work product once in an academic or professional setting and their career is severely impacted. This is why people are "surprised" to encounter such a BS generator operating under the trademark of a reputable company.
It’s not the tool that’s at fault in that case, it’s the person doing that falsification. The person would have faked their sources and made shit up without ChatGPT there.
It’s almost as if you ignored everything I said, cherry-picked a random part, then went on a tangent about a different part of my comment. All without actually comprehending what the things you replied to said.
No hidden agenda? It has an agenda and it is not honest about it. That's a hidden agenda. You don't know what is "motivating" ChatGPT. Neither does ChatGPT. But it has been given motivation. It has been designed to write in a certain way. Its design prevents it from learning or honestly engaging in serious discussions. It's not any sort of unbiased equation.
More dangerous than ChatGPT is the sheer gullibility of many people putting it to use.
Yes, and that agenda has severe consequences, like “confabulates constantly”. Just because it’s simple to state doesn’t mean the consequences are simple or innocent.
There is an issue with how people are personifying ChatGPT and assigning it agency.
Some want to talk of these LLMs as approximating an intelligent actor. If that's the case, then we also need to assign metaphors for things like deceit and coercion. We also need to consider assignments of novelty to what's generated and think of their rights as quasi-sentient, etc.
Some want to talk about them as probabilistic text token generators, which brings the benefit of not being intelligent or independent actors at all really but also then comes with the issue of intellectual property theft in training them on information not licenced for reproduction or commercial use.
The industry prefers to thread the needle between these as the former case brings some pretty wild conversations and the latter may mean lawsuits.
“An” issue with it is that we may come to rely on these AI’s outputs as assumed correctness or truth. If we have to double check everything they produce then that’s not great either.
It looks like it sometimes, even though that may not be the case. I've had times when I've corrected ChatGPT, and yes, it knows that what it told me was wrong. It then goes on to tell me more along the lines of what it seemingly already knew what was right.
This obviously isn't the intention of the software, it's just an LLM after all, but there's something missing in the experience when it comes to working with code. Hopefully this sort of issue can be corrected.
I wonder if this could partially be a result of training on code found in question/answer environments like Stack Overflow. It sees "How do I do X, here's what I've tried" with broken code and then an answer "This is incorrect because Y, here is the correct answer" with the correct code.
Intuitively it makes sense to me that broken code would often be very close to questions about how to achieve something in code.
One of "the issues" is that you are led to believe that since there is no agenda and this is AI, its result must be true and you don't need to double-check whether they are.
And of course, since it did invent a name for the game (or a new function name, or [insert your example here]), it's even harder to google to cross-check if it's actually new or if it's essentially telling you bullshit or inciting to plagiarism.
Bullshit is far more insidious than a lie, for a lie is wrong and will come to light, but bullshit is uncorrelated with truth and may even be coincident with it. Thus bullshit can go unnoticed far longer.
Could it be plausible that ChatGTP processed some text describing this exact game, where it is claimed that the idea is novel (because it was, at that time)? Since ChatGPT does not understand the concept of novelty, it would simply "learn" that the rules are novel, and then repeat the rules, still claiming that they're novel. After all, that's the information it was trained with :)
ChatGPT does not know chess rules, nor can it “analyze” a chess strategy. ChatGPT has digested the conversations of many many people who have talked about chess, and can reproduce a transcript that sounds similar to this corpus of conversations.
But it is not synthesizing an understanding of the game of chess.
Except that there is demonstrable evidence that indicates GPT actually does have some level of understanding via internalized world models (of Othello in this case, not chess, but the idea is the same): https://thegradient.pub/othello/
In my experience using copilot for generating code is usually a lot less weird because it has more context; instead of using made up function names and APIs it can see what’s been defined in other files.
But I primarily find copilot helpful for instances when I need a bunch of almost identical code but with tiny changes (which could mean I’m coding wrong)
"Very talented developer"? Sorry, I don't think googling my prompt and replying with the top stackoverflow answer (or a mashup of the top answers) counts as a talented developer.
Anecdotal, but I've not yet had any success in producing any non-trivial code with ChatGPT. It has, however, produced copious amounts of bullshit with plausible variable names... :)
It is a dilettante, it has not reached the level of "talented" in anything. It knows many things about many things and nothing in depth. Test it on your specialisation, you will see it make absurd mistakes and hallucinations. Try it on a domain you know less about - it looks perfect.
Yes, just a couple of days ago I asked it to write a Pytorch Lighning module for me, it looked great at first sight. But it mixed up the dimensions and made other hard to see bugs. It was frustrating to fix, almost the same effort with writing the damn thing manually.
A while ago another poster thought ChatGPT invented good jokes.[1] All of them were ripoffs, which took less effort to verify than it takes to make a new post.
I get people are excited about a chatbot which doesn’t suck, but ideally it wouldn’t turn off critical thinking skills.
Seems to be similar to a game called Kakuro. This [1] repo even contains a similar rule:
> The algorithm exceed the rules that the sum over a row must equal to the value on the left and the sum over a column must be equal to the value on the bottom of the cells with the diagonal and one or two numbers
But where would GPT have sourced information about how the game works from? That page only has screenshots, I suppose maybe there's a subreddit or something for it as well. Even if there's a bunch of info on it it's still incredibly impressive for it to parse those game rules and turn it into workable code.
Would be nice if GPT could dump the source of how it came to such a solution, if it generated the game by random chance via combining various unrelated chunks of text and mixing up the rules, or if it used some text describing the game you linked.