[azertyvode]

How do you prevent the agent from reward-hacking the hidden eval? e.g. writing training data that effectively leaks the eval distribution rather than teaching a general skill?

[Danau5tin]

Great question. Today the answer is that there is nothing stopping this, but you are right this is a 100% valid reward hacking possibility.

No signs of it in the run I did, but for any future larger scale runs, it would be wise to dedicate some focus to prevent this.