[paraschopra]

Well, isn't it sort of expected?

It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).

What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.

So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.

[andy12_]

I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?

There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?

[valand]

Can the awareness simply come from injecting knowledge of itself during fine tuning and then during a chat a system prompt is injected to add a particular context that triggers its self-knowledge?

[chamomeal]

Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?

[JacobAsmuth]

Would you expect to find concepts related to emotions evoked by thinking those thoughts present? Or meta-descriptions of thoughts? Sounds like you're just post-hoc rationalizing to me.

[krackers]

On the face of it, yes? Emotions are very salient part of text, and as a language model you'd hope that it models them. I think the more surprising finding is that J-space is actually less load bearing than you'd assume, that you can ablate a lot of it and enough of the residual stream structure remains that it still produces coherent text.

That's not to dismiss claims of there being an "inner world" or "conscious experience" (which isn't really a falsifiable claim, the whole p-zombie thing). But purely in terms of _why_ you'd expect J-space to contain those things, given that the j-space is a subspace of the residual stream with coordinates we can interpret, it seems like your priors should be that anything that could help accomplish its pretraining & post-training objectives would be captured in there.

And this also helps provide an explanation of some of their claims they observed. For instance, they way they present J-space ablation seems almost mystical, that ablating j-space suddenly turns a "ensouled" model into a robotic one. But j-space is really just a specific subspace within the residual stream, so ablating j-space is not much different than adding a steering vector. And presumably to ablate j-space they nulled out a lot of those dimensions, which would ikely involve nulling out some of of the concepts related to emotion. So their claim could be rephrased as "injecting a steering vector that removes emotional components, results in the model having a robotic voice".

[JacobAsmuth]

Are emotions actually salient parts of text or is that just because you have a human brain which is tuned to recognize emotions?

You're using "I find it easy to recognize emotions in text therefore it is a simple task", but we know for a fact that some tasks which are easy for humans are hard for LLMs, like counting objects in an image, while other tasks are easy for humans and easy for LLMs, like adding single-digit numbers.

It's not readily apparent to me that precisely modelling emotional state of the characters in a piece of text is the second and not the first, which you seem to assume. In fact, the work as presented seems to indicate that it's a class much closer to the first.

[valand]

I also think this. But more in the sense where both end of the LLM are trained using words through repeating arithmetics, considering LLM itself is a repeating pattern of connections, the space in the middle if extracted the same way as the beginning and the end would become data that make sense to us.

[amelius]

Yeah it would be more surprising if all the hidden state was completely uncorrelated to anything in the output.

[jll29]

"France is a beautiful country" may also still continued by "...in the heart of Europe".