[lanstin]

Have you ever had a professor or mentor walk you through some experiences as a way to help you understand some complex concepts? I guess language is used in that but the ideas are conveyed by the process not the language.

And I rather disagree that mathematics is language, when I did maths there was a distinct difference from understanding a thing and then writing it down.

And modalities that don’t include “where is this body in space” or “I am lonely” don’t seem to capture some essential elements of sensory experience. It is a mistake to separate the interior from the exterior in the analysis of how our thinking works.

I think the folks saying these systems have a kind of intelligence, but a non-human kind are correct - whether that can lead to an independent intelligence that can stay stable sane and focused for weeks or months as many people can, without some sort of embodied cognition providing over all wellness checks to keep the system of thinking sane, remains to be seen. In the meantime, in my hands, they can do very nice dataviz programming to make complex systems much more transparent, even as just throwing all the ops data at them and asking what’s up is, several years in, still not worth doing.

[ben_w]

> And I rather disagree that mathematics is language, when I did maths there was a distinct difference from understanding a thing and then writing it down.

It is generally the case that there's a difference between "understanding a thing" and "writing it down", as demonstrated by every student who studies for the exam, the Chinese Room thought experiment, and Business Bullshit Bingo.

For example, I can copy the next sentence of yours, but I have no idea what:

> And modalities that don’t include “where is this body in space” or “I am lonely” don’t seem to capture some essential elements of sensory experience. It is a mistake to separate the interior from the exterior in the analysis of how our thinking works.

means, can you rephrase that by as much as possible? Preferably without a double negative?

Agree regarding sanity issues of AI. Extremely unlikely we make something "stable" so early in our attempts.

[lanstin]

Our sensory experience as wired into our brains include notions of our own body, both physically “where is the leg now and where is it going” and as a sort of monitor of our brain/body state “hunger” “restlessness” “lonely” “need to talk this torrent of ideas out with someone”.

I was arguing that in:

> With multimodal models, at this point tokens are closer to units of sensory experience.

The tokens are inherently missing some extremely vital pieces of sensory experience, namely the ones that establish the narrative of a self.

[ben_w]

Thanks :)

> The tokens are inherently missing some extremely vital pieces of sensory experience, namely the ones that establish the narrative of a self.

Hmm.

While I agree LLMs are missing many pieces of sensory experience, it is unclear to me which pieces are necessary and sufficient for a narrative of self.

Clinical dissociation (and, at least to the approximation of public stereotypes, Buddhism) come to mind as an example where the presence of usual sensory input is insufficient for a sense of self: https://www.mayoclinic.org/diseases-conditions/dissociative-...

More broadly: while I agree that LLMs are not like us*, I am unclear why this matters in this context?

The behaviour of tokens in a transformer seem to me to behave like sensory input, just not human-like sensory input. The closest human analogy would be if the entire context window was a retina, each rod and cone one of the tokens in that context, and the output was filling in the blind spot (but of course, even this is a very loose analogy).

* even if the engineering teams were trying to do that, which they are not, it would be unlikely to converge on us so soon

[TeMPOraL]

> as demonstrated by every (...) the Chinese Room thought experiment

Wait, did I miss the Chinese Room suddenly becoming sensible, or relevant? Last I checked it was a failure to accept that the system of "human + room" can in fact understand Chinese, even if the components individually don't.

[ben_w]

My point here is that regardless of how you view the entire system of [human + room + books], the human themselves in that system are still just a cog. The human themselves doesn't know Chinese, they are literally writing down Chinese without understanding it, by design of the thought experiment.

"Is the Chinese Room intelligent/conscious?" is a scissor question*, i.e. people strongly disagree about what it shows.

Myself, I'm with you: the system collectively is intelligent, despite the human in the loop being reduced to a cog in that system. The biological analogy to "the human does not understand Chinese therefore the system is not conscious" would be "no individual cell in a human body knows what the human is doing therefore humans are not conscious".

* https://www.lesswrong.com/w/scissors-statements

[TeMPOraL]

> Have you ever had a professor or mentor walk you through some experiences as a way to help you understand some complex concepts? I guess language is used in that but the ideas are conveyed by the process not the language.

Yes, but I believe this is happening with LLMs too. Language (as in text, symbols, actions) is a vehicle, but much like us, language models have internal models and represent concepts (this has been directly, empirically demonstrated few years ago), and they don't "think" in tokens either[0].

> And modalities that don’t include “where is this body in space” or “I am lonely” don’t seem to capture some essential elements of sensory experience. It is a mistake to separate the interior from the exterior in the analysis of how our thinking works.

That's fair. LLMs don't capture every dimension we experience. There's also history - our individual lived experiences since birth are, in my view, something else entirely. It's not a modality, but it's also not something currently possible to capture in or post training.

> I think the folks saying these systems have a kind of intelligence, but a non-human kind are correct - whether that can lead to an independent intelligence that can stay stable sane and focused for weeks or months as many people can, without some sort of embodied cognition providing over all wellness checks to keep the system of thinking sane, remains to be seen.

Possibly. I definitely agree it's not human intelligence. I think it's human-like, in the sense of human-approximating, by virtue of how it's trained[1], but it's arriving there via a different path so end result can still be alien (though I speculate approximation will hold[2]).

> even as just throwing all the ops data at them and asking what’s up is, several years in, still not worth doing.

I guess depends on the complexity of the case (and my understanding of your example); e.g. in my case, I had stellar results from giving Sonnet and Opus (from 4.6 all the way to now) access to my Home Assistant instance. They aren't perfect at making dashboards, but they're excellent at surfacing insights I didn't even realize were possible to get.

--

[0] - That's distinct from the "tokens are units of thinking" heuristic, which still holds for mechanistic reasons - best analogy IMO is "clock signal" in ICs.

[1] - The overall goal function is literally just "generate output that looks sensible to a human", in fully general, unqualified sense. Or, put another way, we're just brute-forcing DWIM, and rating the output by whether it's "what I meant".

[2] - Thinking about constraints on biological evolution, whatever the design of a human mind is, the fundamentals behind it must be so simple, that a greedy incremental optimizer random-walked into it. Given how far we've got with LLMs using simple architecture and crude training methods, and especially how eerily similar their failure modes are to human cognitive failure modes, I suspect LLMs are actually attracted towards the same fundamental design as evolution discovered.

[lanstin]

Not much in biology is simple. The time scales for this random walk and the ruthless energy budget of organisms makes for pretty crazy re-use of structures and chemicals even at relatively simple things.

The time scales and mechanisms of investor driven development don’t seem to share much with evolutionary processes. And digitally there’s not really a native equivalent to the global and dynamic state that the chemical melee inside and between cells provides.

[TeMPOraL]

Not much in biology is simple, but one of the ruthless rules is that an incremental change must be net non-negative in terms of fitness to stick around. In particular, this prevents evolution from keeping around adaptations "for future need", preventing chaining up many individually useless adaptations that become beneficial only when all of them come together. Key term here is "metabolic cost". This reduces the many ways evolution could've cobbled together our minds to a much, much smaller subset of sequences of steps that provided intermediate value along the way.