[mupuff1234]

I think what it shows that it has minimal "understanding" of the problem - otherwise such small variations wouldn't pose a challenge. Training it to handle these specific small variations doesn't change that.

It's good in automation, not understanding.

[Lerc]

If it were a complete failure on variations I would be inclined to agree. Instead it was a 30% drop in performance. I would characterise that as limited understanding.

[cgriswald]

My guess is that what’s understood isn’t various parts of solving the problem but various aspects of the expected response.

I see this more akin to a human faking their way through a conversation.

[CamperBob2]

I see this more akin to a human faking their way through a conversation.

That works in English class. Try it in a math class and you'll get a much lower grade than ChatGPT will.

[sirolimus]

Fully agree with this