[vidarh]

If you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made)

Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometimes pinpoint useful things about how they were prompted.

[amluto]

Humans can do one thing that AI agents are 100% completely incapable of doing: being accountable for their actions.

[jumpconc]

You haven't met certain humans. Not all humans have internal capacity for accountability.

The real meaning of accountability is that you can fire one if you don't like how they work. Good news! You can fire an AI too.

[pessimizer]

Bad news! They will not be aware that you have done this and will not care.

[Zak]

The purpose of firing a person shouldn't be vengeance but to remove someone who is unreliable or not cost effective.

It's similarly reasonable to drop a tool that's unreliable, though I don't think that's a reasonable description here. Instead, they used a tool which is generally known to be unpredictable and failed to sandbox it adequately.

[bigstrat2003]

The purpose of firing a person is to remove someone unreliable, but also, the person having that skin in the game makes him behave more reliably. The latter is something you cannot do with an LLM.

The cold hard fact is: LLMs are an unreliable tool, and using them without checking their every action is extremely foolish.

[lukan]

"The cold hard fact is: LLMs are an unreliable tool, and using them without checking their every action is extremely foolish."

You mean checking every action of theirs outside the sandbox I suppose? Otherwise any attempt at letting an agent do some work I would consider foolish.

[jumpconc]

The AI company has skin in the game which motivates them to produce reliable AIs.

[hun3]

But it's still a bit more difficult to sue them for leaking your company's data.

At least for now.

[unyttigfjelltol]

I disagree. They could fire Claude and their legal counsel could pursue claims (if there were any, idk)-- the accountability model is similar. Anthropic probably promised no particular outcome, but then what employee does?

And in the reverse, if a person makes a series of impulsive, damaging decisions, they probably will not be able to accurately explain why they did it, because neither the brain nor physiology are tuned to permit it.

Seems pretty much the same to me.

[yladiz]

> They could fire Claude and their legal counsel could pursue claims (if there were any, idk)-- the accountability model is similar.

What do you mean by fire? And how is the accountability similar to an employee?

[grey-area]

Don’t forget learning, humans can learn, LLMs do not learn, they are trained before use.

[HighGoldstein]

Do we? Or are we born with pre-training (all the crucial functions the brain does without us having to learn them) and a context window orders of magnitude larger than an LLM?

[compass_copium]

It is incredible how willing and eager AI boosters are to denigrate the incredible miracle of human consciousness to make their chatbots seem so special.

No, we are not born with all the pre-training we need. That is rather the point of education, teaching people's brains how to process information in new, maybe unintuitive ways.

[addedGone]

They learn on the next update :p

[grey-area]

That’s training, not learning.

[quantummagic]

Yup. And eventually there will be online learning, that doesn't require a formal update step. People keep conflating the current implementation, as an inherent feature.

[antonvs]

That’s a feature that other humans impose on whoever’s being held accountable. There’s no reason in principle we couldn’t do the same with agents.

[LPisGood]

How would you fire an agent? This impacts the company that makes the LLM, but not the agent itself.

[lmm]

What does that actually mean in practice? You can yell at human if it makes you feel better, sure, but you can do that with an AI agent too, and it's approximately as productive.

[jeremyccrane]

Yep.

[jayd16]

You might as well be asking a tape recorder why it said something. Why are we confusing the situation with non-nonsensical comparisons?

There is no internal monologue with which to have introspection (beyond what the AI companies choose to hide as a matter of UX or what have you). There is no "I was feeling upset when I said/did that" unless it's in the context.

There is no ghost in the machine that we cannot see before asking.

Even if a model is able to come up with a narrative, it's simply that. Looking at the log and telling you a story.

[vidarh]

Sperry's experiments makes it quite clear that the comparison is not nonsensical: humans can't reliably tell why we do things either. It is not imbuing AI with anything more to recognise that. Rather pointing out that when we seek to imply the gap is so huge we often overestimate our own abilities.

[fluoridation]

Humans at least have a mental state that only they are privy to to work from, and not just their words and actions. The LLM literally cannot possibly have a deeper insight into the root cause than the user, because it can only work from the information that the user has access to.

[lmm]

> Humans at least have a mental state that only they are privy to to work from

Maybe. How do you tell? What would you expect to be different if they didn't?

> The LLM literally cannot possibly have a deeper insight into the root cause than the user, because it can only work from the information that the user has access to.

Insight is not solely a function of available input information. Arguably being able to search and extract the relevant parts is a far more important part of having insights.

[fluoridation]

>Maybe. How do you tell? What would you expect to be different if they didn't?

I think you're asking how I would know if other people were P-zombies. That's an inappropriate question because I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.

Unvoiced thoughts are trickier to prove, but quite often they leave their mark in the person's voiced thoughts.

>Insight is not solely a function of available input information. Arguably being able to search and extract the relevant parts is a far more important part of having insights.

LLMs are notoriously bad at judging relevance. I've noticed quite often if you ask a somewhat vague question they try to cold-read you by throwing various guesses to see which one you latch onto. They're very bad at interpreting novel metaphors, for example.

[lmm]

> I didn't talk about subjective experience, just about internal state. There's no question about whether other people have internal states. I can show someone a piece of information in such a way that only they see it and then ask them to prove that they know it such that I can be certain to an arbitrarily high degree that their report is correct.

Well, sure, but that much is equally true for an LLM with a scratchpad or what have you. (I guess you could say that the user should have access to the LLM's scratchpad and therefore be just as able to understand the state as the LLM itself, but as we move towards the LLM using its own state vectors that's less and less true in practice). I agree that a human may have a mood or secret knowledge or what have you in a way that an LLM wouldn't, but if all you're positing is access to some inert but hidden state then that feels like a Toaster-Enhanced Turing Machine.

[jayd16]

It is non-sensical because you're simply bringing in comparisons without anything linking the two. You might as well be talking about how oranges, and bicycles think as well as that is just as relevant as how humans think in this discussion.

In fact, talking about "thinking" at all is already the wrong direction to go down when trying to triage an incident like this. "Do not anthropomorphize the lawnmower" applies to AI as much as Larry Ellison.

[vidarh]

The thing linking the two is that neither are able to accurately introspect and explain the actual reason why they made a decision.

If thinking is the wrong direction to go down, then it is also the wrong direction to go down when talking about humans.

[jayd16]

If your plane fails to fly and humans can't fly then we should be looking at the musculature of humans when working on the plane?

[abcde666777]

Slight pushback - I think there's still a lot more consistency and coherence in a human's recollection of their motives than an LLM.

Sometimes I think we're too eager to compare ourselves to them.

[vidarh]

We have pretty much evidence to support that human recollection includes the right data to be able to ascertain why we actually did something.

[tempaccount5050]

I think you might be misinterpreting that. I always understood it to mean that when the two hemispheres can't communicate, they'll make things up about their unknowable motivations to basically keep consciousness in a sane state (avoiding a kernel panic?). I don't think it's clear that this happens when both hemispheres are able to communicate properly. At least, I don't think you can imply that this special case is applicable all the time.

[vidarh]

We have no reason to believe it is a special case. The fact that these patients largely functioned normally when you did not create a situation preventing the hemispheres from synchronising suggests otherwise to me. There's no reason to think the ability to just make things up and treat it as if it is truthful recollection would just disappear because there are two halves that can lie instead of just one.

[cmiles74]

None of the developers that I’ve worked with have had the hemispheres of their brains severed. I suspect this is pretty rare in the field.

[lmm]

> None of the developers that I’ve worked with have had the hemispheres of their brains severed.

But are their explanations for how they behaved any more compelling than those of people who have? If so, why?

[pixl97]

This still doesnt stop post ad hoc explanations by humans.

[tempaccount5050]

I feel like your conflating a deep misconfiguration of a brain with lying. These things are completely different.

[layer8]

The thing is, the LLM mostly just states what it did, and doesn't really explain it (other than "I didn't understand what I was doing before doing it. I didn't read Railway's docs on volume behavior across environments."). Humans are able of more introspection, and usually have more awareness of what leads them to do (or fail to do) things.

LLMs are lacking layers of awareness that humans have. I wonder if achieving comparable awareness in LLMs would require significantly more compute, and/or would significantly slow them down.

[vidarh]

Sperry's experiments suggests we don't have that awareness, but think we do as our brains will make up an explanation on the spot.

[pierrekin]

I agree that the model can help troubleshoot and debug itself.

I argue that the model has no access to its thoughts at the time.

Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something.

If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context it recorded (chain of thought for example).

[XenophileJKO]

Anthropic's introspection experiments have seemed to show that your argument is falsifiable.

https://www.anthropic.com/research/introspection

[sumeno]

> In fact, most of the time models fail to demonstrate introspection—they’re either unaware of their internal states or unable to report on them coherently.

You got the wrong takeaway from your link.

[XenophileJKO]

The parent said: "I argue that the model has no access to its thoughts at the time."

This is falsified by that study, showing that on the frontier models generalized introspection does exist. It isn't consistent, but is is provable.

"no access" vs. "limited access"

[sumeno]

There is no way for a user to know whether the LLM has introspection in a given case or not, and given that the answer is almost always no it is much better for everyone to assume that they do not have introspection.

You cannot trust that the model has introspection so for all intents and purposes for the end user it doesn't.

[dwheeler]

I would say "limited and unreliable access". What it says is the cause might be the cause, but it's not on any way certain.

[fragmede]

Claude code and codex both hide the Chain of Thought (CoT) but it's just words inside a set of <thinking> tags </thinking> and the agent within the same session has access to that plaintext.

[fc417fc802]

Those are just words inside arbitrary tags, they aren't actually thoughts. Think of it as asking the model to role play a human narrating his internal thought process. The exercise improves performance and can aid in human understanding of the final output but it isn't real.

[antonvs]

Why do you believe that humans have access to an “internal thought process”? I.e. what do you think is different about an agent’s narration of a thought process vs. a human’s?

I suspect you’re making assumptions that don’t hold up to scrutiny.

[fc417fc802]

I made no such claim and I don't understand what direct relevance you believe the human thought process has to the issue at hand.

You appear to be defaulting to the assumption that LLMs and humans have comparable thought processes. I don't think it's on me to provide evidence to the contrary but rather on you to provide evidence for such a seemingly extraordinary position.

For an example of a difference, consider that inserting arbitrary placeholder tokens into the output stream improves the quality of the final result. I don't know about you but if I simply repeat "banana banana banana" to myself my output quality doesn't magically increase.

[antonvs]

> I don't understand what direct relevance you believe the human thought process has to the issue at hand.

You're the one who raised it. Perhaps you should clarify what you mean by "isn't real" - do you believe a human narrating their thought process is saying something that's more real?

Someone else replied to your comment asking essentially the same question, perhaps better phrased:

> What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?

[DiogenesKynikos]

Given that LLMs can speak basically any language and answer almost any arbitrary question much like a human would, the claim that LLMs have comparable (not identical) thought processes to humans does not seem extraordinary at all.

[yladiz]

Are you legitimately arguing that humans don’t have an internal thought process in some way?

[vidarh]

They're arguing that we have no evidence that humans have access to our underlying thoughts any more than the models do.

[lmm]

What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?

[fc417fc802]

I ask a human "predict what a mouse would do here". In an effort to understand why the prediction is sometimes wrong I ask "walk me through what the imaginary mouse is thinking". Upon examination I exclaim "aha! there's the error" but sadly it's not actually because the output prediction was not based on the thinking trace in any robust manner.

That's a loose analogy but it fails to fully illustrate the degree of decoupling here. For example the weirdness of LLM performance being increased via the output of empty sequences.

[lmm]

> I ask a human "predict what a mouse would do here". In an effort to understand why the prediction is sometimes wrong I ask "walk me through what the imaginary mouse is thinking". Upon examination I exclaim "aha! there's the error" but sadly it's not actually because the output prediction was not based on the thinking trace in any robust manner.

Is this meant to be an analogy for a human or an LLM? Where would it be different in the other case?

[jmalicki]

It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt.

[fc417fc802]

It's important to be aware that while those "thoughts" can be a useful aid for human understanding they don't seem to reliably reflect what's going on under the hood. There are various academic papers on the matter or you can closely inspect the traces of a more logically oriented question for yourself and spot impossible inconsistencies.

[mmoll]

It doesn’t mean that these “thoughts” influenced their final decision the way they would in humans. An LLM will tell you a lot of things it “considered” and its final output might still be completely independent of that.

[jmalicki]

Its output quite literally is not independent, as the "thinking tokens" are attended to by the attention mechanism.

[grey-area]

They do not in fact do that. The ‘thoughts’ are not a chain of logic.

[sumeno]

You have a fundamental misunderstanding of what the model is doing. It's not your fault though, you're buying into the advertising of how it works

[eleumik]

Those are a funny progress bar made by a micro model , is just ui

[emp17344]

That is absolutely not what the split brain experiment reveals. Why would you take results received from observing the behavior of a highly damaged brain, and use them to predict the behavior of a healthy brain? Stop spreading misinformation.

[nuancebydefault]

Such 'highly damaged' brain is still 90 percent or more structured the same as a normal human brain. See it as a brain that runs in debug mode.

It is known that the narrative part of the brain is separate from the decision taking brain. If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains.

[Jensson]

> If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains.

Yes brains can hallucinate reasons, doesn't mean they always do. If all reasons given were hallucinations then introspection would be impossible, but clearly introspection do help people.

[vidarh]

Because said "highly damaged brain" in most respects still functions pretty much like a healthy one.

There is no misinformation in what I wrote.