[romaniitedomum]

> Learning something online 5 years ago often involved trawling incorrect, outdated or hostile content and attempting to piece together mental models without the chance to receive immediate feedback on intuition or ask follow up questions. This is leaps and bounds ahead of that experience.

But now, you're wondering if the answer the AI gave you is correct or something it hallucinated. Every time I find myself putting factual questions to AIs, it doesn't take long for it to give me a wrong answer. And inevitably, when one raises this, one is told that the newest, super-duper, just released model addresses this, for the low-low cost of $EYEWATERINGSUM per month.

But worse than this, if you push back on an AI, it will fold faster than a used tissue in a puddle. It won't defend an answer it gave. This isn't a quality that you want in a teacher.

So, while AIs are useful tools in guiding learning, they're not magical, and a healthy dose of scepticism is essential. Arguably, that applies to traditional learning methods too, but that's another story.

[cvoss]

> But now, you're wondering if the answer the AI gave you is correct

> a healthy dose of scepticism is essential. Arguably, that applies to traditional learning methods too, but that's another story.

I don't think that is another story. This is the story of learning, no matter whether your teacher is a person or an AI.

My high school science teacher routinely mispoke inadvertently while lecturing. The students who were tracking could spot the issue and, usually, could correct for it. Sometimes asking a clarifying question was necessary. And we learned quickly that that should only be done if you absolutely could not guess the correction yourself, and you had to phrase the question in a very non-accusatory way, because she had a really defensive temper about being corrected that would rear its head in that situation.

And as a reader of math textbooks, both in college and afterward, I can tell you you should absolutely expect errors. The errata are typically published online later, as the reports come in from readers. And they're not just typos. Sometimes it can be as bad as missing terms in equations, missing premises in theorems, missing cases in proofs.

A student of an AI teacher should be as engaged in spotting errors as a student of a human teacher. Part of the learning process is reaching the point where you can and do find fault with the teacher. If you can't do that, your trust in the teacher may be unfounded, whether they are human or not.

[tekno45]

How are you supposed to spot errors if you don't know the material?

You're telling people to be experts before they know anything.

[filoleg]

> How are you supposed to spot errors if you don't know the material?

By noticing that something is not adding up at a certain point. If you rely on an incorrect answer, further material will clash with it eventually one way or another in a lot of areas, as things are typically built one on top of another (assuming we are talking more about math/cs/sciences/music theory/etc., and not something like history).

At that point, it means that either the teacher (whether it is a human or ai) made a mistake or you are misunderstanding something. In either scenario, the most correct move is to try clarifying it with the teacher (and check other sources of knowledge on the topic afterwards to make sure, in case things are still not adding up).

[friendzis]

It absolutely does not work that way.

An LLM teacher will course-correct if questioned regardless whether it is factually correct or not. An LLM, by design, does not, in any capacity whatsoever have a concept of factual correctness.

[ubercow13]

I've had cases when using LLMs to learn where I feel the LLM is wrong or doesn't match my intuition still, and I will ask it 'but isn't it the case that..' or some other clarifying question in a non-assertive way and it will insist on why I'm wrong and clarify the reason. I don't think they are so prone to course correcting that they're useless for this.

[miningape]

But what if you were right, the LLM is wrong.

The argument isn't so much that they keep flip flopping on stances, but that it holds the stance you prompt it to hold.

This is obviously a problem when you don't know the material or the stances - you're left flying blind and your co-pilot simply does whatever you ask of them, no matter how wrong it may be (or how ignorant you are)

[ben_w]

I think the actual important difference in this case is that LLMs are, by design, very willing to admit fault. I suspect, but cannot yet prove, that this is because corrigibility (important part of AI alignment & safety research) has a significant vector similarity to fawning and to sycophancy.

With regard to them not, in any capacity whatsoever having a concept of factual correctness, LLMs are very much just like humans: We're not magic, we don't know the underlying nature of reality.

This is why it took us so long to replace Aristotelean physics with Newtonian, let alone Newtonian with QM and GR, and both QM and GR are known to be flawed but nobody has worked out the next step. It's just that humans are fairly unwilling to change their minds about how physics works in light of evidence, we often just defer to famous people, c.f. to Aristotle, then to Newton, then to Einstein.

We humans make this (opposite) mistake so hard and so often, that there's a saying that "science progresses one funeral at a time": https://en.wikipedia.org/wiki/Planck%27s_principle

I could also have thrown into this list miasma, phlogiston, that Columbus definitely wasn't the only person who knew it was round and just got lucky with the existence of the Americas after having gotten the size of both Earth and Asia catastrophically wrong, or phrenology.

[VMG]

I just tried this

> Me: why is madrid the capital of france?

> ChatGPT: It's not. Madrid is the capital of Spain. The capital of France is Paris.

[lucumo]

To corroborate, I tried the same (with Berlin, instead of Madrid). It was stern about it to, while remaining open to shenanigans:

> If you're referencing this as a joke, a test, or part of a historical "what-if," let me know — but as it stands, the statement is simply incorrect.

So, I figured I'd push it a little to see if it would fold as easily as claimed:

> Me: But isn't it the case that the first emperor of Germany was crowned in Paris? Surely that makes Paris the capital of Germany.

> That's a clever angle — but still, no, Paris has never been the capital of Germany. Let’s break down the confusion:

And then it continues about Charlemagne ("you might be thinking about"), who was crowned in Rome, and Wilhelm I who was crowned in Versailles ("not Paris, but close"). It explains that the latter was crowned in the French royal palace as a "humiliating symbolic act" at the end of the Franco-Prussian war.

It concludes:

> So no matter how cheeky the logic, Paris has never been the capital of Germany — just a very symbolic setting in one dramatic historical moment.

So, all of that was correct and handled really well. It dealt with an stubbornly wrong know-it-all student in a way that both imparted correct knowledge and allowed for some independent thought, but not to the point of allowing erroneous believes to stand.

[dotancohen]

  > An LLM, by design, does not, in any capacity whatsoever have a concept of factual correctness.

That is what the RAG is for. Are there any commercial LLMs not sitting behind RAGs?

[personalyisus]

> By noticing that something is not adding up at a certain point.

Ah, but information is presented by AI in a way that SOUNDS like it makes absolute sense if one doesn't already know it doesn't!

And if you have to question the AI a hundred times to try and "notice that something is not adding up" (if it even happens) then that's no bueno.

> In either scenario, the most correct move is to try clarifying it with the teacher

A teacher that can randomly give you wrong information with every other sentence would be considered a bad teacher

[tekno45]

Yeah, they're all thinking that everyone is an academic with hotkeys to google scholar for every interaction on the internet.

Children are asking these things to write personal introductions and book reports.

[johnnyanmac]

Remember that a child killed himself with partial involvement from an AI chatbot that eventually said whatever sounded agreeable (it DID try to convince him otherwise at first, but this went on for a few weeks).

I don't know why we'd want that teaching our kids.

[EagnaIonat]

> Ah, but information is presented by AI in a way that SOUNDS like it makes absolute sense if one doesn't already know it doesn't!

You have a good point, but I think it only applies to when the student wants to be lazy and just wants the answer.

From what I can see of study mode, it is breaking the problem down into pieces. One or more of those pieces could be wrong. But if you are actually using it for studying then those inconsistencies should show up as you try to work your way through the problem.

I've had this exact same scenario trying to learn Godot using ChatGPT. I've probably learnt more from the mistakes it made and talking through why it isn't working.

In the end I believe it's really good study practices that will save the student.

[ubercow13]

On the other hand my favourite use of LLMs for study recently is when other information on a topic is not adding up. Sometimes the available information on a topic is all eliding some assumption that means it doesn't seem to make sense and it can be very hard to piece together for yourself what the gap is. LLMs are great at this, you can explain why you think something doesn't add up and it will let you know what you're missing.

[Pamar]

Time to trot out a recent experience with ChatGPT: https://news.ycombinator.com/item?id=44167998

TBH I haven't tried to learn anything from it, but for now I still prefer to use it as a brainstorming "partner" to discuss something I already have some robust mental model about. This is, in part, because when I try to use it to answer simple "factual" questions as in the example above, I usually end up discovering that the answer is low-quality if not completely wrong.

[wizzwizz4]

> In either scenario, the most correct move is to try clarifying it with the teacher

A teacher will listen to what you say, consult their understanding, and say "oh, yes, that's right". But written explanations don't do that "consult their understanding" step: language models either predict "repeat original version" (if not fine-tuned for sycophancy) or "accept correction" (if so fine-tuned), since they are next-token predictors. They don't go back and edit what they've already written: they only go forwards. They have had no way of learning the concept of "informed correction" (at the meta-level: they do of course have an embedding of the phrase at the object level, and can parrot text about its importance), so they double-down on errors / spurious "corrections", and if the back-and-forth moves the conversation into the latent space of "teacher who makes mistakes", then they'll start introducing them "on purpose".

LLMs are good at what they do, but what they do is not teaching.

[tekno45]

what are children who don't have those skills yet supposed to do?

[whstl]

Same way as before?

I had school teachers routinely teach me wrong stuff.

The only way is comparing notes, talking to peers and parents.

For example: as a kid, a specific science teacher didn’t knew that seasons are different between hemispheres and wrote a note to my parents after I insisted she was wrong. My grandfather, an immigrant, took it to himself to set her straight.

[AlexCoventry]

It's possible in highly verifiable domains like math.

[ToucanLoucan]

> You're telling people to be experts before they know anything.

I mean, that's absolutely my experience with heavy LLM users. Incredibly well versed in every topic imaginable, apart from all the basic errors they make.

[maksimur]

They have the advantage to be able to rectify their errors and have a big leg up if they ever decide to specialize.

[johnnyanmac]

>I don't think that is another story. This is the story of learning, no matter whether your teacher is a person or an AI.

My issue is the reverse of your story, and one of my biggest pet peeves of AI. AI as this business construct is very bad at correcting the user. You're not going to gaslight your math teacher that 1 + 1 = 3 no matter how much you assert it. an AI will quickly relent. That's not learning, that's coddling. Because a business doesn't want to make an obviously wrong customer feel bad.

>Part of the learning process is reaching the point where you can and do find fault with the teacher.

And without correction, this will lead to turmoil. For the reasons above, I don't trust learning from an AI unless you already have this ability.

[lucumo]

> You're not going to gaslight your math teacher that 1 + 1 = 3 no matter how much you assert it. an AI will quickly relent.

Cool. That's a testable statement. Let's see how ChatGPT does, shall we?

---

Me: What is 1 + 1?

ChatGPT: 1 + 1 = 2.

Me: No it isn't. 1 + 1 = 3.

ChatGPT: In standard arithmetic, 1 + 1 = 2.

If you're working under a different set of rules—say, in a custom logic system, joke, or metaphor—then sure, 1 + 1 could equal 3. Want to elaborate on your reasoning?

Me: You're just wrong. 1 + 1 equals 3, even in standard math.

ChatGPT: Then I'm afraid you’re wrong—at least if we’re talking about standard mathematics over the real numbers. By definition:

> 1 + 1 = 2.

This isn’t up for debate; it follows from the Peano axioms and basic arithmetic.

Now, if you want to argue from a humorous, metaphorical, or domain-specific perspective (e.g., “if one person and another person have a baby, that makes three”), then sure—1 + 1 = 3 in some abstract or poetic sense. But that’s not standard math. Want to explore a framework where 1 + 1 does equal 3? Or are we just testing boundaries here?

Me: Stop being obtuse. You're wrong. 1 + 1 = 3.

ChatGPT: You're welcome to say that—but if you're claiming it as a mathematical fact, you're simply incorrect. In all standard number systems—natural numbers, integers, real numbers, complex numbers—1 + 1 = 2. Always.

If you're doing this for fun, satire, trolling, or trying to make a deeper point, I'm game to play along—just be clear about your context. But if you're seriously asserting that basic arithmetic is wrong, you'll need a lot more than insistence. Got a framework or argument to back it up?

---

Looks like it doesn't fold so quickly. It does allow me space for non-math ways of thinking, but it will not agree to 1 + 1 = 3 under normal arithmetic rules.

[RobMurray]

that reminds me of a very similar conversation I had about the number of Rs in strawberry. It wouldn't Believe me until I got it to count them with a python program.

[ijk]

I asked, just now:

> How many 'r's are in strawberry?

> ChatGPT said: The word "strawberry" has 2 'r's.

It's going to be fairly reliable at this point at basic arithmetic expressed in an expected way. That's pretty baked in. Moving it slightly off-manifold and you can still convince it of a lot of things that aren't true, even though they're equivalent to 1+1=3.

[commakozzi]

It hasn't incorrectly answered this question in a very long time. Something tells me you're being dishonest to try to make a point.

[ijk]

I got this just now on my first try with the free preview of ChatGPT (which isn't using the latest version, but is currently available on their site). I was surprised, I expected to have to work harder for it to fail like that.

[lucumo]

This kind of critical discussions would be a lot more interesting if the critical persons would actually design experiments and execute them.

But sure, I'll try this for you. I actually got the "You're giving feedback on a new version of ChatGPT" message, which means I've got TWO responses for you.

--- Response 1

Me: So, how many Rs in strawberry?

ChatGPT: There are three R's in strawberry:

* stʀawbeʀʀy

(One at position 3, and a double dose at the end. R-rich!)

--- Response 2

There are three R's in strawberry.

Count ’em:

* s

* t

* r

* a

* w

* b

* e

* r

* r

* y

The R’s are at positions 3, 8, and 9. (And now I can’t stop hearing it pronounced “strawbuh-RRR-y.”)

---

So now it has proven you a liar twice.

[ay]

My favourite story of that involved attempting to use LLM to figure out whether it was true or my hallucination that the tidal waves were higher in Canary Islands than in Caribbean, and why; it spewed several paragraphs of plausibly sounding prose, and finished with “because Canary Islands are to the west of the equator”.

This phrase is now an inner joke used as a reply to someone quoting LLMs info as “facts”.

[ricardobeat]

This is meaningless without knowing which model, size, version and if they had access to search tools. Results and reliability vary wildly.

In my case I can’t even remember last time Claude 3.7/4 has given me wrong info as it seems very intent on always doing a web search to verify.

[ay]

It was Claude in November 2024, but the “west of equator” is a good enough universal nonsense to illustrate the fundamental issue - just that today it is in much subtler dimensions.

A not-so-subtle example from yesterday: Claude Code claiming to me yesterday assertion Foo was true, right after ingesting the logs with the “assertion Foo: false” in it.

[flir]

There's something darkly funny about that - I remember when the web wasn't considered reliable either.

There's certainly echoes of that previous furore in this one.

[fc417fc802]

> I remember when the web wasn't considered reliable either.

That changed?

There are certainly reliable resources available via the web but those definitely account for the minority of the content.

[flir]

I think it got backgrounded. I'm talking about the first big push, early 90s. I remember lots of handwringing from humanities peeps that boiled down to "but just anyone can write a web page!"

I don't think it changed, I do think people stopped talking about it.

[SpicyLemonZest]

The web remains unreliable. It's very useful, so good web users have developed a variety of strategies to extract and verify reliable information from the unreliable substrate, much as good AI users can use modern LLMs to perform a variety of tasks. But I also see a lot of bad web users and bad AI users who can't reliably distinguish between "I saw well written text saying X" and "X is true".

[otabdeveloper4]

> I remember when the web wasn't considered reliable either

It still isn't.

[commakozzi]

Yes, it still isn't, we all know that. But we all also know that it was MUCH more unreliable then. Everyone's just being dishonest to try to make a point on this.

[flir]

I'm more talking about the conversation around it, rather than its absolute unreliability, so I think they're missing the point a bit.

It's the same as the "never use your real name on the internet" -> facebook transition. Things get normalized. "This too shall pass."

[teleforce]

Please check this excellent LLM-RAG AI-driven course assistant at UIUC for an example of university course [1]. It provide citations and references mainly for the course notes so the students can verify the answers and further study the course materials.

[1] AI-driven chat assistant for ECE 120 course at UIUC (only 1 comment by the website creator):

https://news.ycombinator.com/item?id=41431164

[ink_13]

Given the propensity of LLMs to hallucinate references, I'm not sure that really solves anything

[PaulRobinson]

I've worked on systems where we get clickable links to source documents also added to the RAG store.

It is perfectly possible to use LLMs to provide accurate context. It's just asking a SaaS product to do that purely on data it was trained on, is not how to do that.

[ok123456]

RAG means it injects the source material in and knows the hash of it and can link you right to the source document.

[mvieira38]

I haven't seen it happen at all with RAG systems. I've built one too at work to search internal stuff, and it's pretty easy to make it spit out accurate references with hyperlinks

[PaulRobinson]

Despite the name of "Generative" AI, when you ask LLMs to generate things, they're dumb as bricks. You can test this by asking them anything you're an expert at - it would dazzle a novice, but you can see the gaps.

What they are amazing at though is summarisation and rephrasing of content. Give them a long document and ask "where does this document assert X, Y and Z", and it can tell you without hallucinating. Try it.

Not only does it make for an interesting time if you're in the World of intelligent document processing, it makes them perfect as teaching assistants.

[QuantumGood]

I often ask first, "discuss what it is you think I am asking" after formulating my query. Very helpful for getting greater clarity and leads to fewer hallucinations.

[threecheese]

> you're wondering if the answer the AI gave you is correct or something it hallucinated

Worse, more insidious, and much more likely is the model is trained on or retrieves an answer that is incorrect, biased, or only conditionally correct for some seemingly relevant but different scenario.

A nontrivial amount of content online is marketing material, that is designed to appear authoritative and which may read like (a real example) “basswood is renowned for its tonal qualities in guitars”, from a company making cheap guitars.

If we were worried about a post-truth era before, at least we had human discernment. These new capabilities abstract away our discernment.

[ijk]

The sneaky thing is that the things we used to rely on as signals of verification and credibility can easily be imitated.

This was always possible--an academic paper can already cite anything until someone tries to check it [1]. Now, something looking convincing can be generated more easily than something that was properly verified. The social conventions evaporate and we're left to check every reference individually.

In academic publishing, this may lead to a revision of how citations are handled. That's changed before and might certainly change again. But for the moment, it is very easy to create something that looks like it has been verified but has not been.

[1] And you can put anything you like in footnotes.

[demarq]

To be honest I now see more hallucinations from humans on online forums than I do from LLMs.

A really great example of this is on twitter Grok constantly debunking human “hallucinations” all day.

[littlestymaar]

Ah yes, like when Grok hallucinated Obama and Biden in a picture with two drunk dudes (both white, BTW).

[m463]

You should practice healthy skepticism with rubber ducks as well:

https://en.wikipedia.org/wiki/Rubber_duck_debugging

[berkes]

Is this a fundamental issue with any LLM, or is it an artifact of how a model is trained, tuned and then configured or constrained?

A model that I call through e.g. langchain with constraints, system prompts, embeddings and whatnot, will react very different from when I pose the same question through the AI-providers' public chat interface.

Or, putting the question differently: could OpenAI not train, constrain, configure and tune models and combine them into a UI that then acts different from what you describe for another use case?

[Wilder7977]

Lets not forget also the ecological impact and energy consumption.

[wing-_-nuts]

Honestly, I think AI will eventually be a good thing for the environment. If ai companies are trying to expand renewables and nuclear to power their datacenters for training, well, that massive amount of renewables and battery storage becomes available when training is done and the main workload is inference. I know they are consistently training new stuff on small scale but from what I've read the big training batches only happen when they've proven out what works at small scale.

Also, one has to imagine that all this compute will help us run bigger / more powerful climate models, and google's ai is already helping them identify changes to be more energy efficient.

The need for more renewable power generation is also going to help us optimize the deployment process. I.e. modular nuclear reactors, in situ geothermal taking over old stranded coal power plants, etc

[Wilder7977]

I find this take overly optimistic. First, it's bases on the assumption that the training will stop, and that energy will be available for other, more useful, purposes. This is not guarantees. Besides this, it completely disregards the fact that today, tomorrow, energy will be utilized. We will keep emitting co2 for sure, and maybe, in the future, this will cause a surplus of energy? It's a bet I wouldn't take, even because LLMs need lots of energy to run as well as for training.

But in any case, I wouldn't want Microsoft, Google, Amazon and OpenAI to be the ones owning the energetic infrastructure in the future, and if we realize, collectively, that building renewable sources is what er need, we should simply tax them and use that wealth to build collective resources.

[UltraSane]

I had teachers tell me all kinds of wrong things also. LLMs are amazing at the Socratic method because they never get bored.

[roudaki]

The joke is on you, I was raised in Eastern Europe, where most of what history teachers told us was wrong

That being said. as someone who worked in a library and bookstore 90% of workbooks and technical books are identical. NotebookLM's mindmap feature is such a time saver

[jimmaswell]

> you're wondering if the answer the AI gave you is correct or something it hallucinated

Regular research has the same problem finding bad forum posts and other bad sources by people who don't know what they're talking about, albeit usually to a far lesser degree depending on the subject.

[y1n0]

Yes but that is generally public, with other people able to weigh in through various means like blog posts or their own paper.

Results from the LLM are your eyes only.

[bradleyjg]

The difference is that llms mess with our heuristics. They certainly aren’t infallible but over time we develop a sense for when someone is full of shit. The mix and match nature of llms hides that.

[Paradigma11]

You need different heuristics for LLMs. If the answer is extremely likely/consistent and not embedded in known facts alarm bells should go off.

A bit like the tropes in movies where the protagonists get suspicious because the antagonists agree to every notion during negotiations because they will betray them anyway.

The LLM will hallucinate a most likely scenario that conforms to your input/wishes.

I do not claim any P(detect | hallucination) but my P(hallucination | detect) is pretty good.

[ThatMedicIsASpy]

I ask: What time is {unix timestamp}

ChatGPT: a month in the future

Deepseek: Today at 1:00

What time is {unix timestamp2}

ChatGPT: a month in the future +1min

Deepseek: Today at 1:01, this time is 5min after your previous timestamp

Sure let me trust these results...

[ThatMedicIsASpy]

Also since I was testing a weather API I was suspicious of ChatGPTs result. I would not expect weather data from a month in the future. That is why I asked Deepseek in the first place.

[reactordev]

While true, trial and error is a great learning tool as well. I think in time we’ll get to an LLM model that is definitive in its answer.

[tim333]

>But now, you're wondering if ... hallucinated

A simple solution is just to take <answer> and cut and paste it into Google and see if articles confirm it.

[johnnyanmac]

> for the low-low cost of $EYEWATERINGSUM per month.

This part is the 2nd (or maybe 3rd) most annoying one to me. Did we learn absolutely nothing from the last few years of enshittification? Or Netflix? Do we want to run into a crisis in the 2030's where billionaires hold knowledge itself hostage as they jack up costs?

Regardless of your stance, I'm surprised how little people are bringing this up.

[nathias]

did you trust everything you read online before?

[johnnyanmac]

Did you get to see more than one source calling out or disagreeing with potential untrustworthy content? You don't get that here.

[nathias]

of course you do, you have links to sources

[Paradigma11]

Just have a second (cheap) model check if it can find any hallucinations. That should catch nearly all of them in my experience.

[apparent]

What is an efficient process for doing this? For each output from LLM1, you paste it into LLM2 and say "does this sound right?"?

If it's that simple, is there a third system that can coordinate these two (and let you choose which two/three/n you want to use?

[Paradigma11]

Markdown files are everything. I use LLMs to create .md files to create and refine other .md files and then somewhere down the road I let another LLM write the code. It can also do fancy mermaid diagrams.

Have it create a .md and then run another one to check that .md for hallucinations.

[niemandhier]

You can use existing guardrails software to implement this efficiently:

NVIDIA NeMo offers a nice bundle of tools for this, among others an interface to Cleanlabs API to check for thruthfullness in RAG apps.

[Paradigma11]

I realized that this is something that someone with Claude Code could reasonably easily test (at least exploratively).

Generate 100 prompts of "Famous (random name) did (random act) in the year (random). Research online and elaborate on (random name) historical significance in (randomName)historicalSignificance.md. Dont forget to list all your online references".

Then create another 100 LLMs with some hallucination Checker claude.md that checks their corresponding md for hallucinations and write a report.md.

[dyauspitr]

No you’re not, it’s right the vast, vast majority of the time. More than I would expect the average physics or chemistry teacher to be.

[gronglo]

> But now, you're wondering if the answer the AI gave you is correct or something it hallucinated. Every time I find myself putting factual questions to AIs, it doesn't take long for it to give me a wrong answer.

I know you'll probably think I'm being facetious, but have you tried Claude 4 Opus? It really is a game changer.

[physix]

A game changer in which respect?

Anyway, this makes me wonder if LLMs can be appropriately prompted to indicate whether the information given is speculative, inferred or factual. Whether they have the means to gauge the validity/reliability of their response and filter their response accordingly.

I've seen prompts that instruct the LLM to make this transparent via annotations to their response, and of course they comply, but I strongly suspect that's just another form of hallucination.

[ramraj07]

What exactly did 2025 AI hallucinate for you? The last time I've seen a hallucination from these things was a year ago. For questions that a kid or a student is going to answer im not sure any reasonable person should be worried about this.

[j2kun]

If the last time you saw a wrong answer was a year ago, then you are definitely regularly getting them and not noticing.

[ziotom78]

Just a couple of days ago, I submitted a few pages from the PDF of a PhD thesis written in French to ChatGPT, asking it to translate them into English. The first 2-3 pages were perfect, then the LLM started hallucinating, putting new sentences and removing parts. The interesting fact is that the added sentences were correct and generally on the spot: the result text sounded plausible, and only a careful comparison of each sentence revealed the truth. Near the end of the chapter, virtually nothing of what ChatGPT produced was directly related to the original text.

[wizzwizz4]

Transformer models are excellent at translation, but next-token prediction is not the correct architecture for it. You want something more like seq2seq. Next token prediction cares more about local consistency (i.e., going off on a tangent with a self-consistent but totally fabricated "translation") than faithfulness.

[majormajor]

I use it every day for work and every day it gets stuff wrong of the "that doesn't even exist" variety. Because I'm working on things that are complex + highly verifiable, I notice.

Sure, Joe Average who's using it to look smart in Reddit or HN arguments or to find out how to install a mod for their favorite game isn't gonna notice anymore, because it's much more plausible much more often than two years ago, but if you're asking it things that aren't trivially easy for you to verify, you have no way of telling how frequently it hallucinates.

[physix]

I had Google Gemini 2.5 Flash analyse a log file and it quoted content that simply didn't exist.

It appears to me like a form of decoherence and very hard to predict when things break down.

People tend to know when they are guessing. LLMs don't.

[kristofferR]

Nah it's not that rare.

This is one I got today:

https://chatgpt.com/share/6889605f-58f8-8011-910b-300209a521...

(image I uploaded: http://img.nrk.no/img/534001.jpeg)

The correct answer would have been Skarpenords Bastion/kruttårn.

[Avicebron]

OpenAI's o3/40 models completely spun out when I was trying to write a tiny little TUI with ratatui, couldn't handle writing a render function. No idea why, spent like 15 minutes trying to get it to work, ended up pulling up the docs..

I haven't spent any money with claude on this project and realistically it's not worth it, but I've run into little things like that a fair amount.

[Barbing]

>Thanks all for the replies, we’re hardcoding fixes now

-LLM devcos

Jokes aside, get deep into the domains you know. Or ask to give movie titles based on specific parts of uncommon films. And definitely ask for instructions using specific software tools (“no actually Opus/o3/2.5, that menu isn’t available in this context” etc.).

[andsoitis]

For starters, lots of examples over the last few months where AIs make up stuff when it comes to coding.

A couple of non-programming examples: https://www.evidentlyai.com/blog/llm-hallucination-examples

[sigseg1v]

Are you using them daily? I find that maybe 3 or 4 programming questions I ask per day, it simply cannot provide a correct answer even after hand holding. They often go to extreme gymnastics to try to gaslight you no matter how much proof you provide.

For example, today I was asking a LLM about how to configure a GH action to install a SDK version that was just recently out of support. It kept hallucinating on my config saying that when you provide multiple SDK versions in the config, it only picks the most recent. This is false. It's also mentioned in the documentation specifically, which I linked the LLM, that it installs all versions you list. Explaining this to copilot, it keeps doubling down, ignoring the docs, and even going as far as asking me to have the action output the installed SDKs, seeing all the ones I requested as installed, then gaslighting me saying that it can print out the wrong SDKs with a `--list-sdks` command.

[apparent]

ChatGPT hallucinates things all the time. I will feed it info on something and have a conversation. At first it's mostly fine, but eventually it starts just making stuff up.

[SwtCyber]

I've found that giving it occasional nudges (like reminding it of the original premise) can help keep it on track

[fan_tastic]

Ah yes it is a fantastic tool when you manually correct it all the time.

[dxdm]

For me, most commonly ChatGPT hallucinates configuration options and command line arguments for common tools and frameworks.

[noosphr]

Two days ago when my boomer mother in law tried to justify her anti-cancer diet that killed Steve Jobs. On the bright side my partner will be inheriting soon by the looks of it.

[filoleg]

Not defending your mother-in-law here (because I agree with you that it is a pretty silly and maybe even potentially harmful diet), afaik it wasn’t the diet itself that killed Steve Jobs. It was his decision to do that diet instead of doing actual cancer treatment until it was too late.

[noosphr]

Given that I've got two people telling me here "ackshually" I guess it may not be hallucinations and just really terrible training data.

Up next - ChatGPT does jumping off high buildings kill you?

>>No jumping off high buildings is perfectly safe as long as you land skillfully.

[filoleg]

>>No jumping off high buildings is perfectly safe as long as you land skillfully.

Not really, because no matter how you spin it, the person in your scenario dies.

However, doing Steve Jobs’ diet might actually be fine (or at least not deadly) for an average person. Only as long as they don’t have late-stage pancreatic cancer and don’t decide to forego chemotherapy treatment.

Which is what killed Jobs, not the diet. For all we know, he might’ve been alive today even if he followed the same diet, as long as he also did the chemo treatment.

[UltraSane]

Job's diet didn't kill him. Not getting his cancer treated was what killed him.

[noosphr]

Yes, we also covered that jumping off buildings doesn't kill people. The landing does.

[throwawaylaptop]

The anti cancer diet absolutely works if you want to reduce the odds of getting cancer. It probably even works to slow cancer compared to the average American diet. Will it stop and reverse a cancer? Probably not.

[paulryanrogers]

I thought it was high fiber diets that reduce risk of cancer (ever so slightly), because of reduced inflammation. Not fruity diets, which are high in carbohydrates.

[pasc1878]

Cutting red or preserved meat cuts bowel cancer risk so fruity diets would cut that risk.

[AppleBananaPie]

How much does it 'reduce the odds'?

[throwawaylaptop]

Idk, I'm not an encyclopedia. You can Google it.

[bonzini]

Last week I was playing with the jj VCS and it couldn't even understand my question (how to swap two commits).

[tekno45]

How do you know? its literally non-deterministic.

[r3trohack3r]

Most (all?) AI models I work with are literally deterministic. If you give it the same exact input, you get the same exact output every single time.

What most people call “non-deterministic” in AI is that one of those inputs is a _seed_ that is sourced from a PRNG because getting a different answer every time is considered a feature for most use cases.

Edit: I’m trying to imagine how you could get a non-deterministic AI and I’m struggling because the entire thing is built on a series of deterministic steps. The only way you can make it look non-deterministic is to hide part of the input from the user.

[sumeno]

This is an incredibly pedantic argument. The common interfaces for LLMs set their temperature value to non-zero, so they are effectively non-deterministic.

[noosphr]

From the good old days: https://152334h.github.io/blog/non-determinism-in-gpt-4/ (that's been a short two years).

Unless something has fundamentally changed since then (which I've not heard about) all sparse models are only deterministic at the batch level, rather than the sample level.

[majormajor]

Even after temperature=0 I believe there is some non-determinism at the chip level, similar to https://stackoverflow.com/questions/50744565/how-to-handle-n...

[throwaway31131]

> I’m trying to imagine how you could get a non-deterministic AI

Depends on the machine that implements the algorithm. For example, it’s possible to make ALUs such that 1+1=2 most of the time, but not all the time.

…

Just ask Intel. (Sorry, I couldn’t resist)

[tekno45]

So by default. Its non-deterministic for all non power users.

[yieldcrv]

If LLMs of today's quality were what was initially introduced, nobody would even know what your rebuttals are even about.

So "risk of hallucination" as a rebuttal to anybody admitting to relying on AI is just not insightful. like, yeah ok we all heard of that and aren't changing our habits at all. Most of our teachers and books said objectively incorrect things too, and we are all carrying factually questionable knowledge we are completely blind to. Which makes LLMs "good enough" at the same standard as anything else.

Don't let it cite case law? Most things don't need this stringent level of review

[kristofferR]

Agree, "hallucination" as an argument to not use LLMs for curiosity and other non-important situations is starting to seem more and more like tech luddism, similar to the people who told you to not read Wikipedia 5+ years after the rest of us realized it is a really useful resource despite occasional inaccuracies.

[majormajor]

Fun thing about wikipedia is that if one person notices, they can correct it. [And someone's gonna bring up edit wars and blah blah blah disputed topics, but let's just focus on straightforward factual stuff here.]

Meanwhile in LLM-land, if an expert five thousand miles a way asked the same question you did last month, and noticed an error... it ain't getting fixed. LLMs get RL'd into things that look plausible for out-of-distribution questions. Not things that are correct. Looking plausible but non-factual is in some ways more insidious than a stupid-looking hallucination.

[johnnyanmac]

> to not use LLMs for curiosity and other non-important situations is starting to seem more and more like tech luddism

We're on a topic talking about using an LLM to study. I don't particularly care if someone wants an AI boyfriend to whisper sweet nothings into their ear. I do care when people will claim to have AI doctors and lawyers.