A Professor Fact-Checked the “Bing It On Challenge." Bing Lost.

Future Tense
The Citizen's Guide to the Future
Oct. 2 2013 6:08 PM

A Professor Fact-Checked the “Bing It On Challenge." Bing Lost.

You're a Bing man, man. Or are you?
You're a Bing man, man. Or are you?

Screenshot / YouTube

Microsoft’s “Bing It On Challenge” is a clever marketing conceit. In a series of commercials, the company bet random passersby that they’d choose Bing’s web-search results over Google’s in a “side-by-side, blind-test comparison.” The results, Microsoft finds, are pretty conclusive: By a two-to-one ratio, people choose Bing. The commercials close by inviting the viewer to “join the 5 million people who visited the challenge at Bing.com.”

Will Oremus Will Oremus

Will Oremus is Slate's senior technology writer.

That makes it sound like the “two-to-one” claim is based on a sample size of 5 million. But Ian Ayres, a professor at Yale Law School, found that hard to believe. So he decided to look into it a little further. What he found does not make Microsoft's claims look particularly solid.


First, the “two-to-one” figure comes not from the 5 million people who have taken the challenge online, but from a separate study of just 1,000 participants. As for the online challenge, Microsoft says it isn't actually keeping track of the results. So Ayres got four of his law students together and ran a study of his own.

Between January and March, the team hired 1,000 subjects on Mechanical Turk and asked them to take the “Bing It On” challenge on Microsoft’s own website. Some were asked to use randomly selected popular search terms, others came up with their own search terms, and a third group used the search terms suggested by Microsoft on the site. The findings:

Our sample group generally preferred Google to Bing analyzed at both the respondent level (53% to 41%) and the individual search level (49% to 42%). The preference for Google was most pronounced when respondents used popular search terms or selected their own search terms. Respondents who used Bing-suggested search terms preferred Bing and Google in nearly equal numbers.

Whoa: Not only did the subjects not prefer Bing two-to-one, they generally preferred Google. The one exception was when they used only the search terms Microsoft suggested—which implies that Microsoft is suggesting search terms more favorable to Bing than Google.

In a blog post on Freakonomics, Ayres concluded that “several of Microsoft’s claims are a little fishy.” In fact, he wrote, “we think that Google has a colorable deceptive advertising claim against Microsoft.”

That seems a little strong. Microsoft can still defend itself by pointing to its own 1,000-person study, which was conducted by a professional third-party research company called Answers Research to guard against bias. It also ran a second 1,000-person study using a different set of search terms, and Bing came out on top again, though by a narrower margin. (That may be why one of Microsoft’s more recent “Bing It On” ads dropped the “two-to-one” claim and just said that people “prefer” Bing over Google.) Still, Ayres’ starkly different findings do raise some questions about how Microsoft’s studies managed to turn up such Microsoft-friendly findings.

I asked Microsoft for comment, and they sent the following statement from Matt Wallaert, a behavioral scientist at Bing:

The professor’s analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100% accurate and we’re glad to see we’ve nudged Google into improving their results. Bing it On is intended to be a lightweight way to challenge peoples’ assumptions about which search engine actually provides the best results. Given our share gains, it’s clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service.

At this point, it's Microsoft's study against Ayres'. That said, Microsoft might have a stronger case if it provided some more substantive criticism of Ayres' methodology than simply asserting that it's "flawed"—and then backing up and downplaying the whole thing as "lightweight." Funny: I didn't see the word "lightweight" in Microsoft's previous write-ups of the study results, let alone those national TV ads.

Update, Thursday, Oct. 3, 10:39 a.m.: Microsoft's behavioral psychologist, Matt Wallaert, has put up a blog post offering a more full-throated defense of the company's "Bing It On Challenge" and the results it claims in its commercials. In the post, Wallaert argues that Microsoft's survey sample of 1,000 was in fact more statistically powerful than the sample used by Ayres. And he defends Microsoft's policy of not tracking the answers given by people taking the challenge on its website. Doing so, he says, would be both unethical and totally unscientific. You can read Wallaert's full post here.

Future Tense is a partnership of SlateNew America, and Arizona State University.



The Ebola Story

How our minds build narratives out of disaster.

The Budget Disaster That Completely Sabotaged the WHO’s Response to Ebola

PowerPoint Is the Worst, and Now It’s the Latest Way to Hack Into Your Computer

The Shooting Tragedies That Forged Canada’s Gun Politics

A Highly Unscientific Ranking of Crazy-Old German Beers


Welcome to 13th Grade!

Some high schools are offering a fifth year. That’s a great idea.


The Actual World

“Mount Thoreau” and the naming of things in the wilderness.

Want Kids to Delay Sex? Let Planned Parenthood Teach Them Sex Ed.

Would You Trust Walmart to Provide Your Health Care? (You Should.)

  News & Politics
The World
Oct. 22 2014 2:05 PM Paul Farmer Says Up to Ninety Percent of Those Infected Should Survive Ebola. Is He Right?
Business Insider
Oct. 22 2014 2:27 PM Facebook Made $595 Million in the U.K. Last Year. It Paid $0 in Taxes
Dear Prudence
Oct. 23 2014 6:00 AM Monster Kids from poorer neighborhoods keep coming to trick-or-treat in mine. Do I have to give them candy?
  Double X
The XX Factor
Oct. 22 2014 10:00 AM On the Internet, Men Are Called Names. Women Are Stalked and Sexually Harassed.
  Slate Plus
Tv Club
Oct. 22 2014 5:27 PM The Slate Walking Dead Podcast A spoiler-filled discussion of Episodes 1 and 2.
Brow Beat
Oct. 22 2014 10:39 PM Avengers: Age of Ultron Looks Like a Fun, Sprawling, and Extremely Satisfying Sequel
Future Tense
Oct. 22 2014 2:59 PM Netizen Report: Twitter Users Under Fire in Mexico, Venezuela, Turkey
  Health & Science
Bad Astronomy
Oct. 23 2014 7:30 AM Our Solar System and Galaxy … Seen by an Astronaut
Sports Nut
Oct. 20 2014 5:09 PM Keepaway, on Three. Ready—Break! On his record-breaking touchdown pass, Peyton Manning couldn’t even leave the celebration to chance.