And all along, the secret was right there in the stochastic gradient descent. On Friday, a team called ” BellKor’s Pragmatic Chaos” declared itself the winner of the $1 million Netflix Prize , the contest to improve the site’s existing movie recommendations formula by 10 percent. We don’t yet know exactly how the likely winners did it—the winning team is keeping quiet, since other contestants still have a month to beat their entry, according to the contest rules. We do know this: The contest probably wasn’t won thanks to a revolutionary idea about how we form our taste in movies.
While such a concept may still be out there, the Netflix Prize champs appear to have notched a major victory for computer science over psychology. That seems like a safe conclusion based on progress reports team members have published over the years. For a taste, try this sentence: “The values of the parameters are learnt by stochastic gradient descent with weight decay on the Probe data.”
I’m sure BellKor’s final paper will contain some insights into the psychology of taste, so long as you’re willing to ponder the math and computing behind it. A 2008 paper (PDF), for example, outlines how to represent the ebb and flow of a movie’s popularity over time, as well as a model for how an individual’s rating technique evolves. ( Cinematch , the model that Netflix uses today, is a simpler formula that correlates lots of data to predict your preferences based on your past ratings.)
As the Times noted over the weekend, the tentative winners are a fusion of four previously independent teams “made up of statisticians, machine learning experts and computer engineers.” The fact that it took the combined powers of four teams to crack the 10 percent ceiling is another telling sign that the Netflix contest won’t produce some elegant, easy-to-parse theory of movie-watching. That shouldn’t diminish the accomplishment. Even if we won’t understand why it works, we should expect real results—that is, smarter recommendations—as soon as the BellKor et al. solution is implemented. The Netflix Prize is a good reminder that the human brain is an extremely powerful computer capable of juggling hundreds of variables, even when it’s thinking about how much it likes Weekend at Bernie’s II .
