[krackers]

Oh I guess another thing related to all of this, is prior work on steering vectors. "Manipulating the j-space" seems not too different from steering, both ultimately work on the residual stream. I think perhaps it makes more sense to think of J-space as just a coordinate system for the residual stream where each coordinate axis is a vocab direction. Compared to vector steering which was much more naive and had to derive the direction via PCA.

I like the clarification from https://x.com/XYHan_/status/2074478449020850623#m

>The “J-space” is not a separate, hidden space. It is an alternative coordinate system for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations

>LLMs think in a subconscious space using tokens narrative is completely misleading because >(1) It's a coordinate system. Not a new/separate space. >(2) Tokens only appear because they specifically built the coordinate system using the unembedding vectors of tokens

There is also a good companion piece by Neel Nanda [1] which answers "Why Jacobians rather than linear regression?" which was another question that came to mind

[1] https://www.lesswrong.com/posts/zFJ3ZdQwrTWE9jT5S/a-review-o...