| I used a slight reworking and got a similar response to you with GPT4. I tried to prime it to think through the possibilities by giving it the context: "This is a difficult problem that many people get wrong. Start by reminding yourself of basic logic rules. Then apply the logic rules to the unicorn situation, considering each unicorn in turn and understanding what it would mean for the rule if the unicorn is turned around. Only after doing that conclude with the unicorns that Tom should turn to have a chance of proving Paul wrong." I gave it this instruction because of other articles I've read where forcing it to give the answer before the reasoning means it gets it wrong more often. It correctly identified that it should use the contrapositive, but still misapplied it, so I gave it that feedback: "your third consideration is a misapplication of the contrapositive. Can you try that case again?" Then it hadn't generated a consideration of the last unicorn (it's possible I was being throttled), so I said: "Consider Unicorn 4 with the contrapositive rule" With those extra pieces of guidance it gave the right answer and for the right reasons. While I was hoping for better, this is still a meaningful improvement over GPT3.5s performance on the same prompt - its answer was so muddled I couldn't see how to coach it. |