[jw1224]

> This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well

I would speculate it’s struggling because of the linear nature of its output, and the red-herring words which crossover between categories.

Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to a certain combination early on will lead to a mistake later.

I expect if you asked it to correct its output in a followup message, it could do so without much difficulty.

[coolness]

> I expect if you asked it to correct its output in a followup message, it could do so without much difficulty.

I had a similar idea to the author and tried this many times, albeit with the free version of ChatGPT. After getting wrong results, I prompted it to correct them, even telling the model explicitly that a category is wrong or doesn't make sense. Nothing I did made a difference.

My two cents on why this doesn't work has to do with the fact that the answer should contain a discrete set of words given in the prompt, and importantly, they should not be duplicated. I suspect that these currents models are not very good at following the instruction "the token should appear in the answer exactly once"

[nigamanth]

> Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to a certain combination early on will lead to a mistake later.

Aren't there already models that CAN look ahead? Or are there none?