[lucideer]

While the thrust of this article is generally correct, I have two issues with it:

1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing.

2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate solution), but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Often, rather than "correcting" itself it merely "teaches" the AI how to be confidently wrong in an amenable & subtle manner which the individual user finds easy to accept (or more difficult to spot).

If anything, unless/until we can solve the (insurmountable) problem of AI being wrong, AI should at least be trained to be confidently & stubbornly wrong (or right). This would also likely lead to better consistency in testing.

[traceroute66]

> is their willingness to correct themselves when asked

Except they don't correct themselves when asked.

I'm sure we've all been there, many, many, many,many,many times ....

   - User: "This is wrong because X"
   - AI: "You're absolutely right !  Here's a production-ready fixed answer"
   - User: "No, that's wrong because Y"
   - AI: "I apologise for frustrating you ! Here's a robust answer that works"
   - User: "You idiot, you just put X back in there"
   - and so continues the vicious circle....

[ACCount37]

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for.

With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix".

Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.

[vidarh]

This is a good point, and to drive this home to people, if you have a conversation of this pattern:

    User: Fix this problem ...
    Assistant: X
    User: No, don't do X
    Assistant: Y
    User: No, Y is wrong too.
    Assistant: X

It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong.

If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further.

If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will.

I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.

[recursive]

This is also a thing that's bad about LLMs. You're holding it wrong if you continue to argue. But LLMs are presented as if we can use the conventions of natural language to communicate with them. That's how they're sold. So if they fail to live up to those expectations, that's still a problem with LLMs.

[vidarh]

It's a problem with LLM's and people are "holding it wrong".

It makes zero difference that they've been sold as doing better if other people learn how to use them effectively and I choose to ignore how to get the best possible results out of them.

[asadotzler]

Except that it's impossible to "hold it right" -- even when following the guidance from its makers.

[xienze]

> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer

Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?

[vidarh]

That's fine once or twice. At that point people should learn that this isn't how they work, and figure out how to use them better.

It's not a tools fault if people insist on continuing to use them in counter-productive ways.

[asadotzler]

It's not the tools fault when people RTFM (guidance from the tool maker) and use it as it's intended (again, by the tool maker, who presumably knows how it works and is in the best position to guide users).

"If you keep pressing the back button like the IE engineers told you to, of course you will fail to go back. To go back you want to press the forward button. Are you an idiot? Press the forward button to go back, at least until the next version release when you will need to press the reload button to go back. Trust me, eventually the back button will go back, but for now only fools press the back button to go back."

[xienze]

They’re non-deterministic, remember? So it’s not always the case that an LLM will get stuck in this sort of loop. Hence why people get frustrated when it happens and continue to think that perhaps it should be working on a more consistent basis.

[psadauskas]

There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant).

If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used.

I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.

[vidarh]

I think in some cases you "just" need to instead up temperature to increase the variety of responses, repeat requests, and use hooks to automatically review and reject bad options.

(And yes, it's a horrible workaround)

[_flux]

Indeed, arguing with LLM is good if you like arguing. For results it's not the way to go.

I think often it's not required to completely start over: just identify the part where it goes off the rails, and modify your prompt just before that point. But yeah, basically the same process.

[vidarh]

Sure, when working with tools - like Copilot for example - that lets you "restore" the conversation to a given point, that has pretty much the same effect. The key is to excise the "bad steps" from the conversation and figure out how to amend the next conversation steps so it doesn't veer off in the wrong direction.

[dingnuts]

I don't know why this could be the case but I have absolutely gotten better results out of the bot after insulting it.

[QuantumGood]

I always ask "Tell me what you think it is I am asking" before asking for a solution. Improves the solution and context.

[rtkwe]

I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.

[stetrain]

Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time.

They tend to very quickly lose useful context of the original problem and stated goals.

[nyeah]

Yes, that is the point of the comment.

[stetrain]

Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment.

Is there anything else I can help you with?

[nyeah]

Ok, fair, clearly I misinterpreted what you wrote.

[therobots927]

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

[gavinray]

  > Yeah I think our jobs are safe.

I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

  > Why doesn’t anyone acknowledge loops like this?

Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

Now what you've got is a junk-soup where the original problem is buried somewhere in the pile.

Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on.

I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.

[dinfinity]

I concur. Something to keep in mind is that it is often more robust to pull an LLM towards the right place than to push it away from the wrong place (or more specifically, the active parts of its latent space). Sidenote: also kind of true for humans.

That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up in the context, the more they do still tend to pull the response towards them even with explicit negation/'avoid' instructions.

I think this is why clearing all the crap from the context save for perhaps a summarizing negative instruction does help a lot.

[gavinray]

  >  positively worded instructions ("do x") work better than negative ones ("don't do y")

I've noticed this.

I saw someone on Twitter put it eloquently: something about how, just like little kids, the moment you say "DON'T DO XYZ" all they can think about is "XYZ..."

[Wowfunhappy]

> That means that positively worded instructions ("do x") work better than negative ones ("don't do y").

In teacher school, we're told to always give kids affirmative instructions, ie "walk" instead of "don't run". The idea is that it takes more energy for a child to figure out what to do.

[fuzzzerd]

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a different solution. No human would keep trying one of the solutions included in the context that are marked as not valid.

[traceroute66]

> No human would keep trying one of the solutions included in the context that are marked as not valid.

Exactly. And certainly not a genius human with the memory of an elephant and a PhD in Physics .... which is what we're constantly told LLMs are. ;-)

[Workaccount2]

I'm sure somewhere in the current labs there are teams that are trying to figure out context pruning and compression.

In theory you should be able to get a multiplicative effect on context window size by consolidating context into it's most distilled form.

30,000 tokens of wheel spinning to get the model back on track consolidated to 500 tokens of "We tried A, and it didn't work because XYZ, so avoid A" and kept in recent context

[gavinray]

  > No human would keep trying one of the solutions included in the context that are marked as not valid.

Yeah, definitely not. Thankfully for my employment status, we're not at "human" levels QUITE yet

[vidarh]

I agree it shouldn't be a problem, but if you don't regularly run into humans who insist on trying solutions clearly signposted as wrong or not valid, you're far luckier than I am.

[ModernMech]

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

This is going to age like "full self driving cars in 5 years". Yeah it'll gain capabilities, maybe it does do 80% of the work, but it still can't really drive itself, so it ultimately won't replace you like people are predicting. The money train assures that AGI/FSD will always be 6-18 months away, despite no clear path to solving glaring, perennial problems like the article points out.

[xienze]

> The money train assures that AGI/FSD will always be 6-18 months away

I vividly remember when some folks from Microsoft come to my school to give a talk at some Computer Science event and proclaimed that yep, we have working AGI, the only limiting factor is hardware, but that should be resolved in about ten years.

This was in 2001.

Some grifts in technology are eternal.

[ForHackernews]

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

How long before there's an AI smart enough to say 'no' to half the terrible ideas I'm assigned?

[more_corn]

Herald AI has a pretty robust mechanism for context cleanup. I think I saw a blogpost from them about it.

[zparky]

Um, how much are you spending on running all these at once?

[RyanOD]

But still under pressure in the short-term, no? As companies lean into AI as a means of efficiency / competitive advantage / cost savings, jobs will be eliminated / reduced while companies find their direction. The potential gains are said to be too big to sit on the sidelines and wait to be a late-adopter.

[therobots927]

Yes hold onto your job like your life depends on it because after this bubble pops the job market will get even worse. Then you need to hold on through the trough until experienced engineers are valued again once all of the AI waste flushes out of the system

[jansper39]

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

[aleph_minus_one]

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

My impression is rather: there exist two kinds of people who are "very invested in this illusion":

1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic.

2. People who have a leftist agenda ("we will soon all be replaced by AI, so politics has to implement [leftist policy measures like UBI]"). If people realize that AI is not so powerful, after all, such leftist political measures whose urgency was argued with the (hypothetical) huge societal changes that will be caused by AI will not have a lot backing in society, or at least not considered to be urgently implemented by society.

[vidarh]

The left is generally extremely sceptical to UBI, as its main proponents tend to be classically liberal groups (so not "US liberal") pushing it as a means to contain and limit welfare systems by dropping welfare programs in favour of a general, low UBI.

The more leftist position ever since the days of Marx has been that "right rather than being equal would have to be unqueal" to be equitable given that people have different needs, to paraphrase from Critique of the Gotha Program - UBI is in direct contradiction to socialist ideals of fairness.

The people I see pushing UBI, on the contrary, usually seems motivated either by the classically liberal position of using it to minimise the state, or driven by a fear of threats to the stability of capitalism. Saving capitalism from perceived threats to itself isn't a particularly leftist position.

[therobots927]

I agree with your first point but regarding your second: I’m as far left as it gets and I don’t think that’s true at all. Most of the influencers I follow despise AI and also are highly skeptical of the outrageous claims made by Sam Altman etc. The reality is that the need for things like universal health care exists today. Tens of millions of people can not get medical care in the US. Insurance companies are allowed to deny claims with no justification. That has nothing to do with AI taking jobs BUT it does involve AI because United Health’s denial rate went through the roof right after they started letting AI determine which claims were covered by policy with no human review. So people on the left are talking about AI in contexts that it doesn’t seem you’re aware of

[vidarh]

Because it's easy to learn to stop engaging with those loops, treating them as a sign you provided too little context, and instead start a new conversation with an expanded prompt.

It doesn't mean these loops aren't an issue, because they are, but once you stop engaging with them and cut them off, they're a nuisance rather than a showstopper.

[asadotzler]

They happen in subtle ways that aren't always easy and are rarely early in a project I want to just throw away.

"So what if you have to throw out a week's worth of work. That's how these things work. Accept it and you'll be happier. I have and I'm happy. Don't you see that it's OK to have your tool corrupt your work half way through. It's the future of work and you're being left behind by not letting your tools corrupt your work arbitrarily. Just start over like a real man."

[vidarh]

Doing a week's worth of work without verifying is unprofessional whether you do it with AI or without.

[traceroute66]

The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong".

But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script).

And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).

[gavinray]

I'd be curious if you could share some poor-performing prompts.

I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.

[mrguyorama]

This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today.

That's not useful.

[dkersten]

My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!

[brookst]

You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong.

You case is no different from:

- AI: "The capital of France is Paris"

- User: "This is wrong, it changed to Montreal in 2005"

- AI: "You're absolutely right! The capital of France is Montreal"

[kelseyfrog]

Instead I get this:

    Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.

[brookst]

I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too.

I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me.

(The module does exist in another project I’m working on)

[kelseyfrog]

Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.

[zero_iq]

I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!

[lucideer]

True. This also often happens.

Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.

[Wowfunhappy]

...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.

[dns_snek]

> but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI.

Yes! I often find myself overthinking my phrasing to the nth degree because I've learned that even a sprinkle of bias can often make the LLM run in that direction even if it's not the correct answer.

It often feels a bit like interacting with a deeply unstable and insecure people pleasing person. I can't say anything that could possibly be interpreted as a disagreement because they'll immediately flip the script, I can't mention that I like pizza before asking them what their favorite food is because they'll just mirror me.

[stingraycharles]

> 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing.

Exactly. One could argue that this is just an artifact from the fundamental technique being used: it’s a really fancy autocomplete based on a huge context window.

People still think there’s actual intelligence in there, while the actual problems by making these systems appear intelligent is mostly algorithms and software managing exactly what goes into these context windows at what place.

Don’t get me wrong: it feels like magic. But I would argue that the only way to recognize a model being “confidently wrong” is to let another model, trained on completely different datasets with different techniques, judge them. And then preferably multiple.

(This is actually a feature of an MCP tool I use, “consensus” from zen-mcp-server, which enables you to query multiple different models to reach a consensus on a certain problem / solution).

[tango12]

The AI being wrong problem is probably not insurmountable.

Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed.

Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not.

If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.

[procaryote]

Depending on your definitions, either:

* there are already lots of "reasoning" models trying meta-cognition, while still getting simple things wrong

or:

* the models aren't doing cognition, so meta-cognition seems very far away

[energy123]

Mechanistic interpretability could play a role here. The sycophancy you describe in chat mode could be when the question is "too difficult" and the AI defaults to easy circuits that rely on simple rule of thumbs (like does the context contain positive words such as "excellent"). The user experiences this as the AI just following basic nudges.

Could real-time observability into the network's internals somehow feed back into the model to reduce these hallucination-inducing shortcuts? Like train the system to detect when a shortcut is being used, then do something about it?

[ninetyninenine]

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before.

But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hype. But inevitably like landing on the moon, humanity is suffering from hype fatigue.

Too much hype makes us numb to the reality of how insane the technology is.

Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap. LLMs cover creativity, comprehension, analysis, knowledge and much more. Hallucinations is it. The final problem is targeted and boxed into something much more narrower then just build a human level AI from scratch.

Don’t get me wrong. Hallucinations are hard. But this being the last thing left is not an underplay. Yes it’s a massive issue but yes it is also a massive achievement to reduce all of agi to simply solving just an hallucination problem.

[NoGravitas]

> Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap.

What you are missing here is that the "hallucinations" you don't like and the "results" you do like are, in terms of the underlying process, exactly the same thing. They are not an aberration you can remove. Producing these kinds of results without "hallucinations" is going to require fundamentally different techniques. It's not a "last gap".

[ninetyninenine]

That's not true. There is we just need to find it.

Humans have a condition called schizophrenia where we literally are incapable of differentiating hallucination and reality. What that capability is, is something we need to find out and discover for both ourselves and LLMs.

For example: Mathematically speaking it's possible to know how far away an inferenced point is away from a cluster of real world data. That delta when fed back into the neural network can allow the LLM to know how speculative a response is. From there we can feed the response back into itself for refinement.

[dns_snek]

You're confounding hallucination in humans, which is a purely sensory experience, and hallucinations in LLMs, which seems to be used to describe every kind of mistake and deficiency in LLMs.

And even if we were to cure schizophrenia in humans, just what makes you think that it would apply to LLMs? Having an extremely weak conceptual model of the world and not being able to reason out of rather simple problems (like LLMs struggle with) isn't schizophrenia.

This oversimplified explanation which posits that neural networks are just like human brains has truly gone too far now.

> Mathematically speaking it's possible to know how far away an inferenced point is away from a cluster of real world data.

And mathematically speaking, how would you accomplish this? As you probably know LLMs don't operate on conceptual ideas, they operate on tokens. That's why LLMs tend to fail when asked to do things that aren't well represented in their training data, they don't have a working model of the world even if they can fake it to a certain degree.

[ninetyninenine]

No there is no confounding. When you hallucinate with schizophrenia you know things that are not true and you sense things that are not true. The hallucinations involve both sensory and knowledge.

A weak conceptual model of the world is the problem. But realize humans also have a weak conceptual model of the world as well and make a bunch of hallucinations based on that weak model. For example many people are still making the claim about LLMs that it’s all stochastic parroting when it’s been proven that it’s not. That is an hallucination. Or the people betting (and not) on the financial success of crypto or AI. We don’t know how either of these things will pan out but people on either team act as if they know definitively. A huge part of human behavior is driven by hallucinations that fill in gaps.

> And mathematically speaking, how would you accomplish this? As you probably know LLMs don't operate on conceptual ideas, they operate on tokens. That's why LLMs tend to fail when asked to do things that aren't well represented in their training data, they don't have a working model of the world even if they can fake it to a certain degree.

It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it. The best you can say is that the LLM has a less accurate approximation of the world than you but ultimately both you and the LLM hold an incorrect model and both you and the LLM regularly hallucinate off of it. You also make up bullshit on things not well represented in your own model.

But like I said we are often (and often not) aware of our own bullshit so providing that to the LLM quantitatively will help it too.

The LLM is not just trained on random tokens it’s trained on highly specific groups of tokens and those groups of represent conceptual ideas. So an LLM is 100 percent trained on concepts and tokens are only an encoding of that concept.

If a group of tokens represents a vector then we can for sure calculate distance between vectors. We know that there are also different types of vectors represented at each layer of the feed forward network that encode reasoning and not just the syntactic order of the tokens.

Like literally there is not very much training data of a human giving instructions to someone to write code and the associated code diff. The fact that an LLM can do this to a useable degree without volumes of similar training data speaks to the fact it knows concepts. This is the same tired argument that has been proven wrong. We already know LLMs aren’t just parroting training data as the majority of the agentic coding operations we currently use LLMs for actually don’t have associated training data to copy.

Given that we know all of these embeddings from the training data (the model had to calculate the embeddings at one point) we can encode proximity and distance into the model via addition and subtraction of the magnitude of vectors and from this we extract a number that ascertains distance between vectors embeddings.

Imagine a best fit 2D curve through a scatter plot of data points. But at the same time that curve has a gradient color along it. Red indicates its very close to existing data points blue indicates its far. We can definitely derive and algorithm that calculates the additional “self awareness” dimension here encoded in color and this can extend to the higher dimensional encoding that is the LLM.

If an LLm is aware that the output is red or blue then it can sort of tell that if the line is blue it’s likely to be an hallucination.

[dns_snek]

> It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it.

I should've said that the model is "missing", not "weak" when talking about LLMs, that was my mistake. Yes I'm a human with an imperfect and in many aspects incorrect conceptual model of the world, that is true. The following aren't real examples, they're hyperbolic to better illustrate the category of errors I'm talking about.

If someone asks me "can I stare into the sun without eye protection", my answer isn't going to change based on how the question is phrased because I conceptually understand that the radiation coming from the sun (and more broadly, intense visible radiation emitted from any source) causes irreversible damage to your eyes, which is a fact stored in my conceptual understanding of the world.

However LLMs will flip flop based on tone and phrasing of your question. Asked normally, they will warn you about the dangers of staring into the sun, but if your question hints at disbelief, they might reply "No you're right, staring into the sun isn't that bad".

I also know that mirrors reflect light, which allows me to intuitively understand that staring at the sun through a mirror is dangerous without being explicitly taught that fact.

If you ask an LLM whether staring into a mirror which is pointed at the sun (oriented such that you see the sun through the mirror) is safe, they might agree that it's safe to do so, even though they "know" that staring into the sun is dangerous, and they "know" that mirrors reflect light. Presumably this is because their training data doesn't explicitly state that staring at a mirror is dangerous.

The way the question is framed can completely change their answer which betrays their lack of conceptual understanding. Those are distinctly different problems. You might say that humans do this too, but we don't call that intelligent behavior, and we tend to have a low opinion of those who exhibit this behavior often.

[indigoabstract]

I think you would have really enjoyed living in the '50s, when the future was bright and colonizing Mars was basically a solved problem.

What we got instead is a bunch of wisecracking programmers who like to remind everyone of the 90–90 rule, or the last 10 percent.

[ninetyninenine]

Mars wasn't part of the 90-90 rule genius. If you know your history, it's mainly because political interest was lost. Technologically sending someone to mars is 100000000x more feasible than agi simply because the technology and theory exists such that we can do it.

AGI is part of the last 10 percent rule. But like that's the entire issue. 90% is still 90% progress. That is massive. And the hype surrounding LLMs has made people forget how far 90% is. People are going, "LLMs are retarded because it has the IQ of a 5 year old". They don't realize how even getting it to the level of a 5 year old was impossible for decades and decades.

[ninetyninenine]

Wisecracking programmers lol. You talk as if programming is like something to be proud of. It’s one of the most lucrative jobs with ease of entry as a boot camp can turn someone from zero to hero in a year.

And then you mouth off a buzz phrase not even coined by a programmer but repeated to the point of annoyance about how the final 10 percent is always the hardest as if programmers who copy the phrase are so smart.

Bro the last 10 percent being the hardest doesn’t mean the previous 90 percent didn’t happen. The first 90 percent is a feat in itself and LLMs can now even do PRs. That was a feat no one just 5 years ago could have predicted was possible in our lifetimes.

Idiot programmers and their generic wise cracks were the ones saying that AI would never be able to pass the Turing test and this was just 4 years ago.

[taco_emoji]

Oh, buddy, LLM hallucinations are not the only gap left for AGI

[ninetyninenine]

It is. After that it's virtually indistinguishable from chatting with a human

[stetrain]

Yes, the quick to correct itself isn't really useful. I would not like a human assistant/intern/pair programmer who when asked how to do X said:

> To accomplish X you can just use Y!

But Y isn't applicable in this scenario.

> Oh, you're absolutely right! Instead of Y you can do Z.

Are you sure? I don't think Z accomplishes X.

> On second thought you're absolutely correct. Y or Z will clearly not accomplish X, but let's try Q....

[sfn42]

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

[indigoabstract]

I think what matters most is that we now know that it's possible, that a computer mimicking most of our abilities (but not all) which we have long considered intelligent is obviously possible in some indeterminate future.

It's not obvious how long until that point or what form it will finally take, but it should be obvious that it's going to happen at some point.

My speculation is that until AI starts having senses like sight, hearing, touch and the ability to learn from experience, it will always be just a tool/help/aider to someone doing a job, but could not possibly replace that person in that job as it lacks the essential feedback mechanisms for successfully doing that job in the first place.

[Workaccount2]

My favorite "paper" on AI pretty accurately describes this line of thinking

https://ai.vixra.org/pdf/2506.0065v1.pdf

[ninetyninenine]

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the initial fever pitched alarmist reaction people had to LLMs when it first came out.

Like you realize humans hallucinate too right? And that there are humans that have a disease that makes them hallucinate constantly.

Hallucinations don’t preclude humans from being “intelligent”. It also doesn’t preclude the LLM from being intelligent.

[dns_snek]

> Your opinion is a minority now and not shared by people on the forefront of building these things.

Minority != wrong, with many historic examples that imploded in spectacular fashion. People at the forefront of building these things aren't immune from grandiose beliefs, many of them are practically predisposed to them. They also have a vested interest in perpetuating the hype to secure their generational wealth.

[ninetyninenine]

It doesn’t but I would argue that evidence is in favor of the majority.

The ai can easily answer correctly complex questions NOT in its data set. If it is generating answers to questions like these out of thin air which fits our colloquial definition of intelligence.

[dns_snek]

LLMs also fail to answer simple questions that require a minimal amount of reasoning which demonstrates that they do not have a working model of the world. Their answers to factual questions will change depending on how you phrase the question, even if the crux of the question is identical:

"Is X true" -> "Yes, X is true."

"Is X a myth?" -> "Yes, X is a myth"

"Is Y a myth?" (where X = Y, rephrased) -> "No, Y is true"

Even when they're provided with all the facts required to reach the correct answer through simple reasoning, they'll often fail to do so.

Worse still, sometimes they can be told what the correct answer is, with a detailed step-by-step explanation, but they'll still refuse to accept it as true, continuing to make arguments which were debunked by the step-by-step explanation.

All state of the art models exhibit this behavior, and this behavior is inconsistent with any definition of intelligence.

[sfn42]

I kind of started this thread but didn't have the energy to argue about it. You provided the exact argument I wanted, thanks for that. This is exactly the reason why I am adamant that LLMs are not intelligence.

They do this really impressive stuff like generate code and hold conversations that makes them seem intelligent, but then they fail at these extremely basic tasks which, to me, proves that it's all an illusion.

It doesn't understand the instructions you give it, it doesn't even understand the answer it gives you. It just consumes and generates tokens. Sure it works pretty well and it's pretty cool stuff, but it's not AI.

[ninetyninenine]

So. Humans hallucinate too and get simple shit wrong all the time too. That’s the current problem with LLMs we know it gets shit wrong. We also know humans get shit wrong and make hallucinatory claims but that doesn’t make us classify humans as not intelligent.

The fact of the matter is that as retarded and as stupid as the LLM is the fact that it’s so prevalent in the world today is because it gets answers right. We ask it things not in its training data and it produces an answer out of a range of possibilities that is to low probability to be produced by ANY other thing other than actual reasoning and logic.

You need to see nuance here and make your assessment of LLMs NOT based on singular aspects of facts. LLMs get shit wrong all the time they also get shit right all the time and so do humans. What does that look like holistically?

Look at the shit it’s getting right . If it’s getting stuff right that’s not in the training data then some mechanism in there is doing actual “thinking” and when it gets shit wrong well, you get shit wrong too. All getting shit wrong does to you is make you a dumbass it doesn’t make you not an intelligent entity. You don’t lose that status as soon as you do something incredibly stupid which I’m sure you’ve done often enough in your life to know the difference.

[CodexArcanum]

LLMS don't "hallucinate" they generate a stochastic sequence of plausible tokens that, in context when read by a human, are a false statement or nonsensical.

They also dont have an internal world model. Well I don't think so, but the debate is far from settled. "Experts" like the cofounders of various AI companies (whose livelihood depends on selling these things) seem to believe that. Others do not.

https://aiguide.substack.com/p/llms-and-world-models-part-1

https://yosefk.com/blog/llms-arent-world-models.html

[ninetyninenine]

I’m not talking about startups with financial stake. I’m talking about academics and researchers who have zero financial stake and are observing the phenomenon. It is utterly clear now that stochastic parroting is not what’s going on.

[kilpikaarna]

> It is clear to everyone who builds LLMs that the AI is intelligent.

So presumably we have a solid, generally-agreed-upon definition on intelligence now?

> autocompleting things with humanity changing intelligent content.

What does this even mean?

[ninetyninenine]

We do it’s fuzzy but we do. You point to a rock all humans say it’s not intelligent. You point to a human all humans say it is intelligent.

Because we can do this, by logic a universally agreed upon definition exists. Otherwise we wouldn’t be able to do this.

Of course the boundaries between what’s not intelligent and what is, is where things are not as universally agreed upon. Which is what you’re referring to and unlike you I am charitably addressing that nuance rather then saying some surface level bs.

The thing is the people who say the LLM (which obviously exists at this fuzzy categorical boundary) is not intelligent will have logical paradoxes and inconsistencies when they examine there own logic.

The whole thing is actually a vocabulary problem as this boundary line is an arbitrary definition given to a made up word that humans created. But one can still say an LLM is well placed in the category of intelligent not by some majority vote but because that placement is the only one that maintains logical consistency with OTHER entities or things all humans place in the intelligent bucket.

For example a lot of people in this thread say intelligence requires actual real time learning, therefore an LLM is NOT intelligent. But then there are humans who literally have anterograde amnesia and they literally cannnot learn. Are they not intelligent? Things like this are inconsistent and it happens frequently when you place LLMs in the not intelligent bucket.

State your reasoning for why your stance is "not intelligent" and I can point out where the inconsistencies lie.

[wila]

that you're arguing with an LLM :)

[ninetyninenine]

he's not.

[NoGravitas]

IOW, the realistic position is not held by the majority of people whose paychecks depend on it being wrong. I'm shocked.

[mrguyorama]

Also people tend to use "Expert" wrong in the AI world. No, your programmer who has ten years experience integrating and deploying ML models is not an "AI Expert", they are programmers with expertise with some libraries. Building a system that integrates with ffmpeg for media conversion does not make you a "Video compression expert".

Go check out anthropic's careers page and see just how few positions even require a formal training in statistics.

Meanwhile I don't see a lot of real statisticians who are that hyped about LLMs. More importantly, it feels like there aren't even that many scientists at the AI companies.

Your average programmer does not have nearly the "question your assumptions and test your beliefs" training that an actual scientist has, which is funny since nearly every bug in code is caused by an assumption you shouldn't have made and should have tested.

[ninetyninenine]

When I say experts. I refer to academia.

You're shocked because you hallucinated an assumption of something I never claimed.

Hallucinations? Does that sound similar to something?

[mcv]

Yeah, but how much of that is wishful thinking? If your job depends on believing this is real intelligence, you're more likely to believe that.

[eCa]

> Like you realize humans hallucinate too right?

A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.

[ninetyninenine]

A person who has schizophrenia and hallucinates to a greater extent than LLMs are clearly defective and not intelligent or sentient.

Because of this we should euthanize all schizophrenics. Just stab them to death or put a bullet in their heads right? I mean they aren’t intelligent or sentient so you shouldn’t feel anything when you do this.

I’m baffled as to why people think of this in terms of PRs. Like the LLM is intelligent but everyone’s like oh it’s not following my command perfectly therefore it’s not intelligent.

[bigstrat2003]

They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn. If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence.

[ninetyninenine]

There’s examples of humans who can’t learn. Have you seen the movie memento.

There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best.

According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of these people one by one with a bullet straight to the head because clearly these people have no actual intelligence or sentience. That’s the implication of your logic.

https://en.m.wikipedia.org/wiki/Anterograde_amnesia

It’s called anterograde amnesia. Do you see how your logic can justify gassing all these people holocaust style?

When I point out the flaw in your logic do you use the new facts to form a new conclusion? Or do you rearrange the facts to maintain support for your existing conclusion?

If you did the later I hate to tell you this, it wasn’t very intelligent. It was biased. But given that you’re human, that’s what you most likely did and it’s normal. But pause for a second and try to do the former of using the new facts to form a different more nuanced conclusion.

[burnte]

I'm just so glad people are seeing this. I started saying this literally days after ChatGPT came out and I started examining the technology. It's SUPER useful, but it's assistive, it can't be trusted to do things autonomously yet. That's ok, though, it can make human workers more productive, rather than worrying about replacing humans.

[KoolKat23]

Gemini 2.5 pro is quite good at being stubborn (well at least the initial release versions, haven't tested since).

[decentrality]

Agreed with #1 ( came here to say that also )

Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) I also agree that LLMs can correct the path being taken, regenerate better, etc...

But the idea that 'AI' needs to be _stubbornly_ wrong ( more human in the worst way ) is a bad idea. There is a fundamental showing, and it is being missed.

What is the context reality? Where is this prompt/response taking place? Almost guaranteed to be going on in a context which is itself violated or broken; such as with `Open Web UI` in a conservative example: Who even cares if we get the responses right? Now we have 'right' responses in a cul-de-sac universe. This might be worthwhile using `Ollama` in `Zed` for example, but for what purpose? An agentic process that is going to be audited anyway, because we always need to understand the code? And if we are talking about decision-making processes in a corporate system strategy... now we are fully down the rabbit hole. The corporate context itself is coming or going on whether it is right/wrong, good/evil, etc... as the entire point of what is going on there. The entire world is already beating that corporation to death or not, or it is beating the world to death or not... so the 'AI' aspect is more of an accelerant of an underlying dynamic, and if we stand back... what corporation is not already stubbornly wrong, on average?

[taco_emoji]

> Pronoun and noun wordplay aside ( 'Their' ... `themselves` )

How is that wordplay? Those are the correct pronouns.