[yorick]

I'm not sure that the answers that the model provides have anything to do with what it's actually doing. The way they seem to be prompting it also exhibits this issue, where they first have it arrive at a conclusion and then come up with an explanation for this conclusion. LLMs do not have an inner voice to reason with, and tokens generated later do not influence earlier tokens (unless you're doing beam search, but you mostly aren't). It would be much improved if asked to do reasoning first and then arrive at a conclusion.