Hacker News new | ask | show | jobs
by mbq 4037 days ago
Still, this also confirms the obvious fact that the selection of the winning team based on that hidden, private test also gives no guarentee that it is not accidently overfitted, especially with tons of submissions. The only reliable option to solve it is to ask participants for self-contained programs and run it inside a CV loop on the organisers' hardware, but this seems too cumbersome to be implemented in reality; I think only TunedIT tries it, but without luck :L