[xpct]

More curiously, why did it feel the incentive to find the solutions? Would its CoT include "the only way to solve this is to download the test set", or would it include "I'd like to inspect a few entries from the test set so I understand the problem better", then inadvertently poisoning itself with the correct answers.

[pixl97]

This is classical reward hacking. For example in school the goal is to pass a test. You can study, something that is hard and takes a lot of time. Or you can steal the answer key, which is risky and can get you in trouble, but may actually be far easier.

There is absolutely no need to prompt the LLM to cheat, they can determine that cheating is an effective method all on their own.