[mtlmtlmtlmtl]

If the person knew how to ask the question "correctly", they wouldn't need to ask.

They asked about systems, not closed systems. ChatGPT just regurgitated the 2nd law, which is just plain wrong no matter how you slice it. Any physicist asked this question would explain the distinction.

[fnordpiglet]

They gave you all the information you needed to ask the next question. It specified “in a closed system,” which meant it made an assumption of the closed system and told you. An observant reader would ask then “and what about an open system?” Or if they’re not that sophisticated, “what about a non-closed system?”

You’ve not talked to many theoretical physicists I see. I’ve not noticed they’re usually that good at explaining things, especially to lay people. They tend to be very good at talking with people at the same level or more advanced, usually in a mathematical form. They tend to be pretty bad at imitating a human.

[mtlmtlmtlmtl]

Yeah, if you understand the field or are observant enough, you can tell the answer is fishy. And if you don't, you can't tell.

So what, anyone who gets a wrong idea from ChatGPT is just unsophisticated and we should ignore it? Why are you so incredibly set on invalidating any criticism of ChatGPT?

You don't see a problem with advertising this LLM as something it isn't? Lots of people seem willing to take ChatGPT completely at face value now, and walk away having learned a bunch of nonsense. And lots of them are smart people, they've just been duped by the hype into thinking LLMs can do things they fundamentally can't.

[fnordpiglet]

Oh I think it should be improved for sure. I just think this is a bad example. I think most of the fact checking can be done using any modern information retrieval system and you can build algorithms that will regenerate answers until they’re factually correct, or use the IR to hint the answer to correctness. We also have very powerful semantic inference engines and other tools that complement LLM output. I think judging the possibilities by the beta is simplistic, and folks are unfairly down on the achievement by picking nits.

[idiotsecant]

In the context of this question, system has a specific technical meaning though. This is a very technical question so chatGPT might be forgiven for assuming the technical meaning.

[mtlmtlmtlmtl]

So what is ChatGPT supposed to be useful for if people need to understand what the answer is in order to ask the "correct" question? Clearly the person didn't pick up on this closed/non-closed distinction which is why they ended up asking on a forum instead of reformulating.

I think GPT just saw "entropy" and "system" and predicted 2nd law. Which is the sort of low effort response you might get if you ask random non-experts on the internet.

[dwringer]

> So what is ChatGPT supposed to be useful for if people need to understand what the answer is in order to ask the "correct" question?

Personally I think there's still a lot of value in mere rephrasing, recontextualizing, perspective shifting, and occasional insightful connection that ChatGPT can do even if it's all using information you already ostensibly know.

As it is often said that one never learns a topic more deeply than when they teach it to others, ChatGPT can serve as the ultimate "rubber duck" coworker for any subject. I'm not sure what we can do about users not using critical thinking, or expecting the machine to do that part for them. There are plenty of non-AI sources of misinformation that can readily be taken uncritically too, though, so it's not necessarily a new problem.

[DonHopkins]

No, rubber ducks are still the ultimate rubber ducks, because they don't talk back with industrial grade overconfident bullshit that misleads and confuses you.

[dwringer]

Is there a body of evidence that suggests people get more misled and confused after using things like ChatGPT? It seems like a reasonable hypothesis, but my own experience doesn't necessarily support it. I've used the language model at character.ai for a bit and have found it to be clarifying in a sense. When the model spits out some overconfident misinformation, it's a great opportunity to argue with the bot about it in ways one could never argue with another person - certainly not a stranger, at least.

Perhaps I've been confused and misled so badly I don't realize it, so all I can really say is I think it's premature to assume people will be any more misled or confused by technologies like ChatGPT when all they have to do now is get on the internet or flip on a TV to be personally targeted with misleading and confusing information already. I think there's very real potential for the technology to give people a lever against misinformation if it helps them understand and explore their own thoughts/thought processes.

I guess to me, fundamentally, it's a question of who's the one with agency over using it, and to what end. I'd be much more comfortable once we can fit models like this on home computers and worry less about them suddenly trying to sell us sponsored products or convince us of some ideology because their creator was paid to do so.