[DougBTX]

> The title is completely accurate.

Accurate, but a half truth. The prize was for a 10% improvement, but before that solution was produced they had already improved by 8.4%. The headline makes it sound like the improvement from zero to 10% was not worth the engineering cost, but really it was the improvement from 8.4% to 10% which cost too much.

[benhamner]

This is why you should award prizes for the top K performers, as opposed to the first one to cross a benchmark.

[Drbble]

And when the teams merge into K groups and then stop trying? Netflix paid for performance, and got performance.

[benhamner]

In the Netflix prize, did all teams merge into one group and stop trying? Nor would we expect so with K groups, especially with exponentially decreasing prize amounts. If this is even slightly a risk, you can compensate by making the maximum number of prizes awarded a function of the number of participating teams. For example, you can award 10 prizes if there are >100 teams, and if not award floor(num_teams/10) prizes, with the prize pool redistributed among the top 10%.

It was pure luck that the threshold for the million-dollar prize was crossed. If it had been arbitrarily set at 11% (as opposed to 10%), then there's a good chance the million dollar prize would have never been paid out.

The advantage of paying out K top prizes is that other teams that don't win outright may have developed additional useful models or insights (or used more computationally efficient algorithms), and you may access to these in this manner.