[krackers]

On the face of it, yes? Emotions are very salient part of text, and as a language model you'd hope that it models them. I think the more surprising finding is that J-space is actually less load bearing than you'd assume, that you can ablate a lot of it and enough of the residual stream structure remains that it still produces coherent text.

That's not to dismiss claims of there being an "inner world" or "conscious experience" (which isn't really a falsifiable claim, the whole p-zombie thing). But purely in terms of _why_ you'd expect J-space to contain those things, given that the j-space is a subspace of the residual stream with coordinates we can interpret, it seems like your priors should be that anything that could help accomplish its pretraining & post-training objectives would be captured in there.

And this also helps provide an explanation of some of their claims they observed. For instance, they way they present J-space ablation seems almost mystical, that ablating j-space suddenly turns a "ensouled" model into a robotic one. But j-space is really just a specific subspace within the residual stream, so ablating j-space is not much different than adding a steering vector. And presumably to ablate j-space they nulled out a lot of those dimensions, which would ikely involve nulling out some of of the concepts related to emotion. So their claim could be rephrased as "injecting a steering vector that removes emotional components, results in the model having a robotic voice".

[JacobAsmuth]

Are emotions actually salient parts of text or is that just because you have a human brain which is tuned to recognize emotions?

You're using "I find it easy to recognize emotions in text therefore it is a simple task", but we know for a fact that some tasks which are easy for humans are hard for LLMs, like counting objects in an image, while other tasks are easy for humans and easy for LLMs, like adding single-digit numbers.

It's not readily apparent to me that precisely modelling emotional state of the characters in a piece of text is the second and not the first, which you seem to assume. In fact, the work as presented seems to indicate that it's a class much closer to the first.