[ahmedfromtunis]

I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.

Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?

I'm not sure J-Space is the answer to that question, but very interesting nevertheless.

[baq]

> I'm now considering the architecture of the service

What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.

[wongarsu]

Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.

There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors

[aabhay]

Its also because the CoT is probably unintelligible

[optimalsolver]

More legible than seems at first glance:

https://www.lesswrong.com/posts/wCSEpT3dTGz4N86Wi/even-illeg...

[orbital-decay]

The text can be legible, but the meaning the model assigns to these words can be subtly different, and you have no way to tell. This is evident when trying to make most modern reasoning models follow a fixed CoT plan by filling a form with placeholders, they're extremely stubborn because they simply don't understand your words in it! They learned their own language in their CoT. Sometimes they're controlled for readability during training but even then they find a way to circumvent this, for example Gemini 3.0 had perfectly readable raw CoT but was barely able to follow such a plan (3.5 Flash is way better at following, they clearly improved it).

[andrewlin247]

> Is the model really "thinking" about that stuff or is just mimicking human "manners"?

Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.

[ACCount37]

The answer is, as is often the case, "yes".

In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.

"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.