[pelorat]

Thought traces are indeed not an accurate representation of what models actually do. If you ask an AI model to add two values it will do so, then in the next prompt ask it to explain the algorithm it used, it will regurgitate that it used some standard textbook method, whilst in reality it used a completely different algorithm. Thinking LLMs don't record the neural pathways they used.