Media & technology
Netflix Pays a Million Dollars for Ten Percent
Netflix and its recommendation team; more than 40,000 competing teams
A 10.06 percent improvement, after almost three years
In 2006 Netflix offered $1 million to anyone who could make its rating predictions 10 percent more accurate. Almost three years and more than 40,000 teams later, two coalitions finished in a dead heat, tied at the four decimal places the rules used. The winning blend of hundreds of models never went into production: the extra accuracy did not justify the engineering. Netflix still called the prize “a great return on investment.” When is a better prediction worth paying for?
At a Glance
- Decision
- Whether a 10 percent better predictor of its members' ratings was worth $1 million, and then whether to put the winners' methods into production
- Method
- An open contest on 100 million ratings, judged on hidden data: half the withheld ratings scored a public leaderboard, the other half, not reported during the contest, decided the prize; matrix factorization and large blends of models (Theorems 13.1 and 13.3, Proposition 13.2)
- Result
- The 10 percent target reached in 2009 by a blend of hundreds of models; two of the 2007 methods put into production, the final blend not, because its gains did not justify the engineering and the company's focus had moved on
Questions for the board
- On what data, untouched by those who built it, has our model been judged (Theorem 13.3)?
- Which decision does the model change, and what is the change worth, net of the cost of running it (Theorem 13.2)?
- If our objective has moved since the model was specified, does its metric still measure value?
- What could our data reveal about individuals, and who has priced that risk before we share it?
Company and organization names identify the organizations whose published decisions the cases discuss. They are trademarks of their respective owners; no affiliation with or endorsement by them is implied.






