[traceroute66]

> is their willingness to correct themselves when asked

Except they don't correct themselves when asked.

I'm sure we've all been there, many, many, many,many,many times ....

   - User: "This is wrong because X"
   - AI: "You're absolutely right !  Here's a production-ready fixed answer"
   - User: "No, that's wrong because Y"
   - AI: "I apologise for frustrating you ! Here's a robust answer that works"
   - User: "You idiot, you just put X back in there"
   - and so continues the vicious circle....

[ACCount37]

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for.

With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix".

Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.

[vidarh]

This is a good point, and to drive this home to people, if you have a conversation of this pattern:

    User: Fix this problem ...
    Assistant: X
    User: No, don't do X
    Assistant: Y
    User: No, Y is wrong too.
    Assistant: X

It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong.

If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further.

If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will.

I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.

[recursive]

This is also a thing that's bad about LLMs. You're holding it wrong if you continue to argue. But LLMs are presented as if we can use the conventions of natural language to communicate with them. That's how they're sold. So if they fail to live up to those expectations, that's still a problem with LLMs.

[vidarh]

It's a problem with LLM's and people are "holding it wrong".

It makes zero difference that they've been sold as doing better if other people learn how to use them effectively and I choose to ignore how to get the best possible results out of them.

[asadotzler]

Except that it's impossible to "hold it right" -- even when following the guidance from its makers.

[vidarh]

I have no problem "holding it right". Just today I had AI write 100% of the code for two different tools, using an AI assistant which wrote all the code for itself after the initial ~100 lines.

It's not hard to learn to be productive with these models.

[xienze]

> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer

Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?

[vidarh]

That's fine once or twice. At that point people should learn that this isn't how they work, and figure out how to use them better.

It's not a tools fault if people insist on continuing to use them in counter-productive ways.

[asadotzler]

It's not the tools fault when people RTFM (guidance from the tool maker) and use it as it's intended (again, by the tool maker, who presumably knows how it works and is in the best position to guide users).

"If you keep pressing the back button like the IE engineers told you to, of course you will fail to go back. To go back you want to press the forward button. Are you an idiot? Press the forward button to go back, at least until the next version release when you will need to press the reload button to go back. Trust me, eventually the back button will go back, but for now only fools press the back button to go back."

[vidarh]

No, it's not the tools fault if you continue to use it in ways that according to you, yourself does not work, despite the availability of better guidance.

Do you always insist on listening to guidance you've observed doesn't work?

It sounds immensely counter-productive.

Meanwhile I'll continue to have AI tools write the majority of my code at this point.

[xienze]

They’re non-deterministic, remember? So it’s not always the case that an LLM will get stuck in this sort of loop. Hence why people get frustrated when it happens and continue to think that perhaps it should be working on a more consistent basis.

[vidarh]

So are people.

It is no more productive to continue to go in circles with an argumentative person who refuses to see reason.

If someone haven't learnt that lesson, they will get poor results at a whole lot more things in life than talking to AI.

[psadauskas]

There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant).

If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used.

I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.

[vidarh]

I think in some cases you "just" need to instead up temperature to increase the variety of responses, repeat requests, and use hooks to automatically review and reject bad options.

(And yes, it's a horrible workaround)

[_flux]

Indeed, arguing with LLM is good if you like arguing. For results it's not the way to go.

I think often it's not required to completely start over: just identify the part where it goes off the rails, and modify your prompt just before that point. But yeah, basically the same process.

[vidarh]

Sure, when working with tools - like Copilot for example - that lets you "restore" the conversation to a given point, that has pretty much the same effect. The key is to excise the "bad steps" from the conversation and figure out how to amend the next conversation steps so it doesn't veer off in the wrong direction.

[dingnuts]

I don't know why this could be the case but I have absolutely gotten better results out of the bot after insulting it.

[QuantumGood]

I always ask "Tell me what you think it is I am asking" before asking for a solution. Improves the solution and context.

[rtkwe]

I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.

[stetrain]

Yep, the LLM will happily continue this spiral indefinitely but I've learned that if providing a bit more context and one correction doesn't provide a good solution, continuing is generally a waste of time.

They tend to very quickly lose useful context of the original problem and stated goals.

[nyeah]

Yes, that is the point of the comment.

[stetrain]

Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment.

Is there anything else I can help you with?

[nyeah]

Ok, fair, clearly I misinterpreted what you wrote.

[therobots927]

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

[gavinray]

  > Yeah I think our jobs are safe.

I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

  > Why doesn’t anyone acknowledge loops like this?

Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

Now what you've got is a junk-soup where the original problem is buried somewhere in the pile.

Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on.

I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.

[dinfinity]

I concur. Something to keep in mind is that it is often more robust to pull an LLM towards the right place than to push it away from the wrong place (or more specifically, the active parts of its latent space). Sidenote: also kind of true for humans.

That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up in the context, the more they do still tend to pull the response towards them even with explicit negation/'avoid' instructions.

I think this is why clearing all the crap from the context save for perhaps a summarizing negative instruction does help a lot.

[gavinray]

  >  positively worded instructions ("do x") work better than negative ones ("don't do y")

I've noticed this.

I saw someone on Twitter put it eloquently: something about how, just like little kids, the moment you say "DON'T DO XYZ" all they can think about is "XYZ..."

[Wowfunhappy]

> That means that positively worded instructions ("do x") work better than negative ones ("don't do y").

In teacher school, we're told to always give kids affirmative instructions, ie "walk" instead of "don't run". The idea is that it takes more energy for a child to figure out what to do.

[fuzzzerd]

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a different solution. No human would keep trying one of the solutions included in the context that are marked as not valid.

[traceroute66]

> No human would keep trying one of the solutions included in the context that are marked as not valid.

Exactly. And certainly not a genius human with the memory of an elephant and a PhD in Physics .... which is what we're constantly told LLMs are. ;-)

[Workaccount2]

I'm sure somewhere in the current labs there are teams that are trying to figure out context pruning and compression.

In theory you should be able to get a multiplicative effect on context window size by consolidating context into it's most distilled form.

30,000 tokens of wheel spinning to get the model back on track consolidated to 500 tokens of "We tried A, and it didn't work because XYZ, so avoid A" and kept in recent context

[gavinray]

  > No human would keep trying one of the solutions included in the context that are marked as not valid.

Yeah, definitely not. Thankfully for my employment status, we're not at "human" levels QUITE yet

[vidarh]

I agree it shouldn't be a problem, but if you don't regularly run into humans who insist on trying solutions clearly signposted as wrong or not valid, you're far luckier than I am.

[ModernMech]

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

This is going to age like "full self driving cars in 5 years". Yeah it'll gain capabilities, maybe it does do 80% of the work, but it still can't really drive itself, so it ultimately won't replace you like people are predicting. The money train assures that AGI/FSD will always be 6-18 months away, despite no clear path to solving glaring, perennial problems like the article points out.

[xienze]

> The money train assures that AGI/FSD will always be 6-18 months away

I vividly remember when some folks from Microsoft come to my school to give a talk at some Computer Science event and proclaimed that yep, we have working AGI, the only limiting factor is hardware, but that should be resolved in about ten years.

This was in 2001.

Some grifts in technology are eternal.

[ForHackernews]

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

How long before there's an AI smart enough to say 'no' to half the terrible ideas I'm assigned?

[more_corn]

Herald AI has a pretty robust mechanism for context cleanup. I think I saw a blogpost from them about it.

[zparky]

Um, how much are you spending on running all these at once?

[RyanOD]

But still under pressure in the short-term, no? As companies lean into AI as a means of efficiency / competitive advantage / cost savings, jobs will be eliminated / reduced while companies find their direction. The potential gains are said to be too big to sit on the sidelines and wait to be a late-adopter.

[therobots927]

Yes hold onto your job like your life depends on it because after this bubble pops the job market will get even worse. Then you need to hold on through the trough until experienced engineers are valued again once all of the AI waste flushes out of the system

[jansper39]

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

[aleph_minus_one]

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

My impression is rather: there exist two kinds of people who are "very invested in this illusion":

1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic.

2. People who have a leftist agenda ("we will soon all be replaced by AI, so politics has to implement [leftist policy measures like UBI]"). If people realize that AI is not so powerful, after all, such leftist political measures whose urgency was argued with the (hypothetical) huge societal changes that will be caused by AI will not have a lot backing in society, or at least not considered to be urgently implemented by society.

[vidarh]

The left is generally extremely sceptical to UBI, as its main proponents tend to be classically liberal groups (so not "US liberal") pushing it as a means to contain and limit welfare systems by dropping welfare programs in favour of a general, low UBI.

The more leftist position ever since the days of Marx has been that "right rather than being equal would have to be unqueal" to be equitable given that people have different needs, to paraphrase from Critique of the Gotha Program - UBI is in direct contradiction to socialist ideals of fairness.

The people I see pushing UBI, on the contrary, usually seems motivated either by the classically liberal position of using it to minimise the state, or driven by a fear of threats to the stability of capitalism. Saving capitalism from perceived threats to itself isn't a particularly leftist position.

[therobots927]

I agree with your first point but regarding your second: I’m as far left as it gets and I don’t think that’s true at all. Most of the influencers I follow despise AI and also are highly skeptical of the outrageous claims made by Sam Altman etc. The reality is that the need for things like universal health care exists today. Tens of millions of people can not get medical care in the US. Insurance companies are allowed to deny claims with no justification. That has nothing to do with AI taking jobs BUT it does involve AI because United Health’s denial rate went through the roof right after they started letting AI determine which claims were covered by policy with no human review. So people on the left are talking about AI in contexts that it doesn’t seem you’re aware of

[vidarh]

Because it's easy to learn to stop engaging with those loops, treating them as a sign you provided too little context, and instead start a new conversation with an expanded prompt.

It doesn't mean these loops aren't an issue, because they are, but once you stop engaging with them and cut them off, they're a nuisance rather than a showstopper.

[asadotzler]

They happen in subtle ways that aren't always easy and are rarely early in a project I want to just throw away.

"So what if you have to throw out a week's worth of work. That's how these things work. Accept it and you'll be happier. I have and I'm happy. Don't you see that it's OK to have your tool corrupt your work half way through. It's the future of work and you're being left behind by not letting your tools corrupt your work arbitrarily. Just start over like a real man."

[vidarh]

Doing a week's worth of work without verifying is unprofessional whether you do it with AI or without.

[traceroute66]

The AI-fanbois will quickly tell you that you are misusing the context or your prompt is "wrong".

But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script).

And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).

[gavinray]

I'd be curious if you could share some poor-performing prompts.

I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.

[mrguyorama]

This is just the new version of "works on my machine". Oh, I was able to contrive a correct answer from my prompt because the random number generator smiled upon me today.

That's not useful.

[dkersten]

My pet peace is when I point out a problem it responds with acknowledgement and then explaining why it’s wrong. Like… I already know why it’s wrong, since I’m the one that pointed it out!

[brookst]

You're conflating "correct themselves" with "are guaranteed to give the correct answer", which are two really different things. And in fact you're just echoing GP's point: their corrections can be wrong.

You case is no different from:

- AI: "The capital of France is Paris"

- User: "This is wrong, it changed to Montreal in 2005"

- AI: "You're absolutely right! The capital of France is Montreal"

[kelseyfrog]

Instead I get this:

    Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.

[brookst]

I was using an oversimplified example to illustrate. In practice, it appears more in large context statements about the context. If the human is wrong, there’s a good chance the AI will cheerfully agree and then be wrong too.

I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me.

(The module does exist in another project I’m working on)

[kelseyfrog]

Help me understand how it's tangibly different from a veteran telling the rookie to find headlight fluid, winter air for the tires, or keys to the bomb range.

[zero_iq]

I've seen ChatGPT get stuck in this loop all by itself, generating a long multi-page answer where it constantly catches itself, refutes itself, offers a new answer with the same problem, rinse and repeat... All in the same response!

[lucideer]

True. This also often happens.

Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.

[Wowfunhappy]

...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.