You probably already know that Amazon’s recommendations can be amusingly off target. (I recently received an e-mail informing me that people who purchased music by that old favorite Original Soundtrack also liked music by Elvis Costello.) Nevertheless, when Amazon recently launched a brand-new tool designed to predict how you’ll like a product—before you buy it, that is—my ears pricked up. It’s one thing for Amazon to tell you how fellow shoppers rated a book they’ve bought and read. But forecasting how much you’ll like it ahead of time? That takes some chutzpah. So I decided to give the feature a test drive.
I started with a relatively clean slate. The site knew what I’d bought and what was waiting in my shopping cart, but since I’d never rated anything, it didn’t know what I liked. On a sheet of paper, I rated the five books I’d read most recently, and then checked my own scores against the ones Amazon predicted for me. Here are my Amazon-forecasted and actual grades, all out of a possible 5 stars:
I duly entered my ratings into the Amazon system, in the hope that it would learn more about my taste. And since I was still getting quite a few entries with no prediction, I decided to beef up the data pool by rating five old favorites, each of which I gave a 5. Here’s how Amazon thought I’d like them:
Not bad. But after tossing out so many top scores, I was afraid the bot would think I loved everything. So I rated five recent books that had either been reviewed well or recommended by friends that I did not like as much as I’d expected.
After rating 15 books, I was still getting a surprising number of items with no rating. What was going on? According to the Help page, the predicted ratings rely on four factors: 1) Items you’ve rated; 2) items you’ve purchased; 3) items you’ve told Amazon you own; 4) comparisons between your activity on the site and that of other customers.
Factor No. 1 makes perfect sense. Nos. 2 and 3 raise a question: If you own it, does Amazon.com assume you liked it? According to Matthew Round, the head of Amazon’s personalization team, yes: “By default we assume you like the things you purchase—otherwise why would you buy them? It’s a very strong declaration of interest in an item to part with money for it.” In order for your ratings to be as accurate as possible, you must go in and rate all your purchases; otherwise the system assumes you like everything you buy. That’s a dangerous assumption for those of us who have plunked down $25 for a new novel only to find that we can’t get past the 10th page.
Round’s explanation of factor No. 4—the comparison of your activity on the site to that of other customers—was somewhat mysterious: “We use your customer profile including your ratings and purchases as a starting point, and given what we know in general about how customers tend to rate the item in question we try to predict how you’ll react.” This method of generating recommendations based on past interests is called collaborative filtering (click here to find out more about how it works).
The accuracy of these predictions will depend on how much data they have on you, as well as how useful the data is in the first place. One weak spot is DVD ratings. Because so few DVDs are released each month compared with books and music (even though they’re comparable in price), people buy a greater proportion of the catalog, which results in less accurate data. With music and books there’s such a massive catalog that a person’s tastes for specific authors and styles can be outlined with reasonable accuracy. When you click on a novel by an author, you tend to see his other works or works by similar novelists. For example, people who bought one of my old favorites, A Supposedly Fun Thing I’ll Never Do Again, by David Foster Wallace, also bought his books Girl With Curious Hair, Brief Interviews With Hideous Men, and Infinite Jest, as well as Dave Eggers’ A Heartbreaking Work of Staggering Genius. I know these to be fine recommendations because I’ve read them all and liked them, too.
The one distorting factor in my experiment was that I’d bought a few books for other people and for research, so there were a few data points in the system obscuring my true likes and dislikes. But cleaning up the data is easy; just check a box next to specific purchases asking that they not count toward your recommendations. Once I’d cleared out gifts and work-related books, I was surprised by the reasonable nature of the recommendations. Amazon recommended several books I’d read and liked: Me Talk Pretty One Day, Infinite Jest, A Gravity’s Rainbow Companion, plus several I hadn’t that sounded tempting—This Is Not a Novel and Open City No. 6.
Still, I had a major hesitation. Presumably Amazon implemented this elaborate rating program to encourage buying. The more you rate, the more accurate the predictions become, and the greater your faith in the prediction system. But unlike a reviewer, Amazon has an incentive to dole out high ratings. Notice that the lowest prediction I got in my initial experiments was a 2.5. If I gave Amazon enough negative feedback on a particular type of book, would it rate something similar a 1? In other words, would Amazon have the guts to tell me that I’d hate a book I was about to buy?
I started my hatchet job with David McCullough, giving 1’s to Truman and his recent John Adams biography (I haven’t actually read these books; I slammed McCullough for the sake of science). I checked another title by McCullough, Mornings on Horseback; no prediction. I gave another terrible rating to another presidential biography, Stephen Ambrose’s Eisenhower: Soldier and President. From there on in, I couldn’t seem to get any predicted ratings on any presidential biographies. Nothing for Jean Edward Smith’s Grant or Barry Schwartz’s Abraham Lincoln and the Forge of National Memory. It seems that instead of giving a bad prediction, Amazon doesn’t give one at all. Wimps!
Then I thought back to my initial experiment and realized that Amazon hadn’t given out any 5’s either. Is Amazon as reluctant to give A’s as it is to give F’s? To find out, I started doling out 5’s to Philip Roth books, starting with the most recent. For The Dying Animal Amazon predicted 3 stars. For The Human Stain it predicted 4. I Married a Communist: 4. American Pastoral: 4. Sabbath’s Theater: up to 4.5. Operation Shylock: 4.5. Finally, by Patrimony, Amazon spit out a 5. Clearly, Amazon is being conservative with its ratings; like any good reviewer, it knows that a surplus of raves will eventually erode its authority.
Anyway, if I had to rate Amazon’s rating system, I’d give it 3 stars. I doubt I’d rely too heavily on it for my book purchases. However, I might use it for entertainment, because it provides a neat keyhole view into the purchasing habits of the anonymous millions. At first it seems odd that people who bought the Guinness Book of Records also bought Chicken Soup for the Preteen Soul. But then I recall spending countless hours in middle school poring over the photos of the world’s heaviest twins and the man smoking hundreds of cigarettes at once. Seeing odd yet plausible connections like this one gives you a certain private detective-ish thrill, as does seeing the logic behind your predicted ratings. (Next to each recommendation you can click a button that will answer the question “Why was I recommended this?” that links to a list of the purchases and ratings they based the recommendation on.) This type of accumulated customer information is of more interest than the ratings that Amazon coughs up. In other words, forget about what I think of the books; I want to know what Amazon thinks of me.
