[HarHarVeryFunny]

I think the BS-generation problem with ChatGPT goes far deeper than citing sources, for a variety of reasons.

1) It's not a search engine, even if it behaves a bit like one. It's not "retrieving answers" to your questions (from sources that it could choose to cite). ChatGPT is really just a "language model", so it has no notion that what you're typing is even a question/query .. your input is just treated as sequence of words (which ChatGPT has zero understanding of), with ChatGPT's response then being a further sequence of words that it has calculated are (one) statistically probable continuation of what you typed (you can keep asking it for alternative answers, and it'll continue generating additional alternative statistically probable continuations).

The websites/etc that ChatGPT was trained on are just sources of language that it consumed in order to learn the statistics that let it make these continuation predictions. It's not memorizing "facts" from websites, just word statistics, and these are mixed in with the statistics from all the other sources it was trained on. If it generates the word "walk" as part of a response, it can't cite a source for that since there essentially is none - only a bazillion text sources it was trained on that collectively made the word "walk" a high probability continuation on the words it had generated leading up to that...

2) Even if ChatGPT had been designed to deal in "facts" (rather that words statistics) associated with specific sources, the bullshit problem isn't just knowing the varied reliability of the sources it was trained on, but how those "facts" are combined. To combine multiple facts and correctly deduce something new from them would require intelligence, but ChatGPT doesn't have any intelligence - it's just a statistical word generator, so the way it combines snippets from different sources is again just statistical word generation, with zero knowledge of the meaning of the words it is generating or whether it makes sense!

What makes ChatGPT seem semi-intelligent is that a lot of what it was trained on was text written by semi-intelligent humans, so the "sequence of words" it is generating, following the statistics of human speech, seems like something a human might say... until you start paying attention to the meaning of the words and realize it's often good-sounding garbage.

[Animats]

Which is the big problem. ChatGPT will produce something reasonable if it's seen good content on the indicated subject. Otherwise, it just makes up plausible blithering, including fake references.

Useful for fiction, advertising copy, and literary criticism. Not so good for fact retrieval.

[larsonnn]

So in other words.

When OpenAI had a way for training live data all big marketing companies would produce a ton of information just to get their „facts“ to ChatGPT.

But as a user you can’t compare different sources like you would do on Google and you only have this BS answer which is fancy but tells you to drink dish cleaner because studies have found out that dish cleaner makes stuff clean and clean is healthy.

[scarface74]

I put in a Python script I wrote to automate some things around AWS. It described the purpose of the scripts. Then I asked it make some changes and it did. I asked it why would I use it. It gave me a plausible explanation. I asked it to add comments and the comments were pretty good.

I even asked it how the script could be improved and it made suggestions around adding error handling and making some hard coded names into command line parameters.

I asked it to give me code to implement the suggestions and it gave me working code.

It’s much better than you give it credit for.

[HarHarVeryFunny]

Sure, depending on what you ask and how that aligns with the content it was trained on and the word statistics it has learned, it can give correct answers.

OTOH I've also asked it what day of the week a given date was and received two different wrong answers depending on the exact phrasing of the question. I've also seen it confidently "explain" why taking 90% of a number and adding 10% of that back will get you to the original number...

The trouble is the output is a mix of truth and lies, and GPT has no way to distinguish between the two.

[scarface74]

I’ll give you that,

I once asked it write a Python script that lists all of the accounts in an AWS organization with a given tag key and value.

It confidently, initiated the SDK (boto3) and the correct object on the SDK (Organizations) and then it called a none existent function - “get_accounts_by_tag”.

The next day I asked it the same question and it got it right using a technique that I would have never thought of.

On the other hand, I asked it “given the following XML file and a DynamoDB table with the following fields, write a Python script that replaces the value node in the file where a corresponding key is found in the table with the value in the value field”.

The code was perfect.

[mr_toad]

> What makes ChatGPT seem semi-intelligent

Its lack of intelligence is not the problem. High intelligence doesn’t preclude misinterpretation, mis-remembering, or overestimating it’s own understanding.

[HarHarVeryFunny]

I think that depends on the goals of ChatGPT and/or what users are hoping to get out of it.

If it was just acting as a search engine using english as the query language, then lack of intelligence wouldn't be an issue - the quality of output would just depend on the quality of the source as we're used to with search engines.

However, what ChatGPT is actually doing - due to it's fundamental nature as a language model (dealing only in word/language statistics) is effectively combining information from multiple sources, which of course is potentially very powerful if it knew HOW to utilize these variously sourced facts to construct a correct answer... but of course it doesn't, so it'll happily generate content mixed from factual and fantasy sources etc, or correct textbook programming exercises with buggy code from beginners it dredged up someplace. It's not just mixed sources though - it's the intelligence of how to take a bunch of raw facts and deduce something from them, and of course ChatGPT is not a deduction engine.

[ilaksh]

If you go to the OpenAI playground and turn down 'temperature' to 0 it seems to not BS at all as far as I can tell.

[HarHarVeryFunny]

Temperature is presumably referring to sampling the output probabilities. With a temperature of 0 it'll be giving you the very highest probability continuation, while with increasingly higher temperatures it'll be sampling from the possible continuations to provide more variety.

In other words, the temperature is controlling the variety of output, but of course doesn't affect what was fed into it in the first place. As the saying goes, Garbage-In, Garbage-Out .. even with a temperature of zero it's still going to be bullshitting since "predict next word" (language model) is fundamentally a bullshitting technology - just keep on spewing out words regardless of meaning.

[Nathanba]

the thing is.. ChatGPT doesn't have to compete with perfectly correct information because the information you search for on Google is often wrong(=SEO spam) too and you have to sift through a lot of garbage or misleading links there too. Sometimes literally, because you get a forum link with a bunch of people saying wrong things and then finally someone says the answer. That's similar to what you have to do on ChatGPT to doublecheck or ask a follow up questions or read more or treat a piece of information with a dose of healthy doubt. Both ChatGPT/Google are very useful and they both produce imperfect results and they both require some human thought.

[marstall]

interesting point about SEO. Does ChatGPT somehow filter out SEO content? Is it rather selective about the domains it crawls? Because Google could certainly turn that switch too - but then it would lose its comprehensiveness ...