Can We Ever Know If Online Advertising Really Works?

Innovation, the Internet, gadgets, and more.
June 17 2014 5:17 PM

We Have No Idea If Online Ads Work

The Internet has given us an ocean of data. Turns out, most of it is pretty useless.

Google founders Larry Page (R) and Sergey Brin (L) speak at a press conference announcing Google's launch of a new transit mapping feature of Google Maps.
Google co-founders Larry Page (right) and Sergey Brin in 2008 in New York City.

Photo by Michael Nagle/Getty Images

In the summer of 2003, Viacom executive Mel Karmazin managed to sum up old media’s horror of the Internet with one of business lore’s greatest vulgar one-liners. Karmazin, a swaggering former ad salesman and onetime CBS Corporation president, had made an expedition to check out Silicon Valley’s hottest young upstart: Google. Sitting in the future search giant’s offices, he listened in dismay as its founders, Larry Page and Sergey Brin, and its CEO, Eric Schmidt, detailed the many ways their company could track and analyze the effectiveness of online advertising. This could not possibly be good for business, Karmazin thought. It had always been nearly impossible for marketers to tell which of their ads worked and which didn’t, and the less they knew, the more a network like CBS could charge for a 30-second spot. Art was far more profitable than science.

Jordan Weissmann Jordan Weissmann

Jordan Weissmann is Slate's senior business and economics correspondent.

As Ken Auletta later recounted in his 2009 history of Google, Karmazin stared at his hosts and blurted out, “You’re f---ing with the magic!”

A decade later, someone finally seems to be, well, messing with Google’s own bag of tricks. Last year, a group of economists working with eBay’s internal research lab issued a massive experimental study with a simple, startling conclusion: For a large, well-known brand, search ads are probably worthless. This month, their findings were re-released as a working paper by the National Bureau of Economic Research and greeted with a round of coverage asking whether Internet advertising of any kind works at all.


“We know a lot less than the advertising industry would like us to think we know,” Steven Tadelis, one of the eBay study’s co-authors, told me.

Ask Google, Facebook, or Twitter, of course, and they’ll reliably bust out third-party research explaining that their ads work just fine, even if consumers don’t always click on them. An entire ecosystem of analytics companies, including big names like ComScore and Nielsen, has evolved to tell clients which online advertisements give them the biggest bang for their bucks. Especially cutting-edge firms, such as Datalogix, have even found ways to draw correlations between the ads consumers see online and what they buy in stores. We are swimming in data. And there are plenty of professionals out there happy to tell corporate America what all that data means, with the help of some fancy mathematical models.

The problem, according to Tadelis and others, is that much of the data websites generate is more or less useless. Some of the problems are practically as old as marketing itself. For instance, companies like to run large ad campaigns during major shopping seasons, like Christmas. But if sales double come December, it’s hard to say whether the ad or the holiday was responsible. Companies also understandably prefer to target audiences they think will like what they’re selling. But that always leads to the nagging question of whether the customer would have gone and purchased the product regardless. Economists call this issue “endogeneity.” Derek Thompson at the Atlantic dubs it the “I-was-gonna-buy-it-anyway problem.”

But the Internet also gunks up attempts at analysis in its own special ways. For instance, if somebody searches for “Amazon, banana slicer,” and clicks on a search ad that pops up right next to his results, chances are he would have made it to Amazon’s site without the extra nudge. Even if he never typed the word Amazon, he still might have gotten to the site through the natural power of search. In the end, it all comes down to the evergreen challenge of distinguishing correlation (e.g., a Facebook user saw an ad and then bought some shoes) from causation (e.g., a Facebook user bought some shoes because he saw an ad).

There is, however, a way to get around these hurdles: Run an experiment. Most analytics companies don’t do that, relying instead on elaborate statistical regressions that try to fix flawed data. But it’s the route Tadelis—now a business professor at the University of California, Berkeley—and his collaborators Tom Blake and Chris Nosko took with eBay. In their first test, the researchers looked at what would happen if the company stopped buying ads next to its own name, which seemed like the most obvious waste of money. To do so, they pulled the ads from Yahoo and MSN but left them running on Google. It turns out, the advertising made virtually no difference. Yet eBay was spending dollars every time a customer clicked an ad instead of the link sitting right below it.



The Ebola Story

How our minds build narratives out of disaster.

The Budget Disaster That Completely Sabotaged the WHO’s Response to Ebola

PowerPoint Is the Worst, and Now It’s the Latest Way to Hack Into Your Computer

The Shooting Tragedies That Forged Canada’s Gun Politics

A Highly Unscientific Ranking of Crazy-Old German Beers


Welcome to 13th Grade!

Some high schools are offering a fifth year. That’s a great idea.


The Actual World

“Mount Thoreau” and the naming of things in the wilderness.

Want Kids to Delay Sex? Let Planned Parenthood Teach Them Sex Ed.

Would You Trust Walmart to Provide Your Health Care? (You Should.)

  News & Politics
Oct. 22 2014 9:42 PM Landslide Landrieu Can the Louisiana Democrat use the powers of incumbency to save herself one more time?
Continuously Operating
Oct. 22 2014 2:38 PM Crack Open an Old One A highly unscientific evaluation of Germany’s oldest breweries.
Gentleman Scholar
Oct. 22 2014 5:54 PM May I Offer to Sharpen My Friends’ Knives? Or would that be rude?
  Double X
The XX Factor
Oct. 22 2014 4:27 PM Three Ways Your Text Messages Change After You Get Married
  Slate Plus
Tv Club
Oct. 22 2014 5:27 PM The Slate Walking Dead Podcast A spoiler-filled discussion of Episodes 1 and 2.
Oct. 22 2014 11:54 PM The Actual World “Mount Thoreau” and the naming of things in the wilderness.
Future Tense
Oct. 22 2014 5:33 PM One More Reason Not to Use PowerPoint: It’s The Gateway for a Serious Windows Vulnerability
  Health & Science
Wild Things
Oct. 22 2014 2:42 PM Orcas, Via Drone, for the First Time Ever
Sports Nut
Oct. 20 2014 5:09 PM Keepaway, on Three. Ready—Break! On his record-breaking touchdown pass, Peyton Manning couldn’t even leave the celebration to chance.