[jonhinson]

Netflix's blog post mentions they used two algorithms from one of the Progress Prize ($50,000) winners.

The article goes on to say "...you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later...We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment."

The title is completely accurate.

[DougBTX]

> The title is completely accurate.

Accurate, but a half truth. The prize was for a 10% improvement, but before that solution was produced they had already improved by 8.4%. The headline makes it sound like the improvement from zero to 10% was not worth the engineering cost, but really it was the improvement from 8.4% to 10% which cost too much.

[benhamner]

This is why you should award prizes for the top K performers, as opposed to the first one to cross a benchmark.

[Drbble]

And when the teams merge into K groups and then stop trying? Netflix paid for performance, and got performance.

[benhamner]

In the Netflix prize, did all teams merge into one group and stop trying? Nor would we expect so with K groups, especially with exponentially decreasing prize amounts. If this is even slightly a risk, you can compensate by making the maximum number of prizes awarded a function of the number of participating teams. For example, you can award 10 prizes if there are >100 teams, and if not award floor(num_teams/10) prizes, with the prize pool redistributed among the top 10%.

It was pure luck that the threshold for the million-dollar prize was crossed. If it had been arbitrarily set at 11% (as opposed to 10%), then there's a good chance the million dollar prize would have never been paid out.

The advantage of paying out K top prizes is that other teams that don't win outright may have developed additional useful models or insights (or used more computationally efficient algorithms), and you may access to these in this manner.

[eli]

Yes, the engineering costs are too hard to justify... now that DVDs by mail is no longer their primary focus. it's not clear to me that that would still be the case if Netflix business were still operating around 2007's constraints.

[akkartik]

Did the entry ranked 5 also have the 2 algorithms? Then the title makes sense. A contest is a good way to motivate the base, but it can also cause lots of effort that is unnecessary in the end.