Can We Ever Know If Online Advertising Really Works?

Innovation, the Internet, gadgets, and more.
June 17 2014 5:17 PM

We Have No Idea If Online Ads Work

The Internet has given us an ocean of data. Turns out, most of it is pretty useless.

Google founders Larry Page (R) and Sergey Brin (L) speak at a press conference announcing Google's launch of a new transit mapping feature of Google Maps.
Google co-founders Larry Page (right) and Sergey Brin in 2008 in New York City.

Photo by Michael Nagle/Getty Images

In the summer of 2003, Viacom executive Mel Karmazin managed to sum up old media’s horror of the Internet with one of business lore’s greatest vulgar one-liners. Karmazin, a swaggering former ad salesman and onetime CBS Corporation president, had made an expedition to check out Silicon Valley’s hottest young upstart: Google. Sitting in the future search giant’s offices, he listened in dismay as its founders, Larry Page and Sergey Brin, and its CEO, Eric Schmidt, detailed the many ways their company could track and analyze the effectiveness of online advertising. This could not possibly be good for business, Karmazin thought. It had always been nearly impossible for marketers to tell which of their ads worked and which didn’t, and the less they knew, the more a network like CBS could charge for a 30-second spot. Art was far more profitable than science.

Jordan Weissmann Jordan Weissmann

Jordan Weissmann is Slate's senior business and economics correspondent.

As Ken Auletta later recounted in his 2009 history of Google, Karmazin stared at his hosts and blurted out, “You’re f---ing with the magic!”

A decade later, someone finally seems to be, well, messing with Google’s own bag of tricks. Last year, a group of economists working with eBay’s internal research lab issued a massive experimental study with a simple, startling conclusion: For a large, well-known brand, search ads are probably worthless. This month, their findings were re-released as a working paper by the National Bureau of Economic Research and greeted with a round of coverage asking whether Internet advertising of any kind works at all.


“We know a lot less than the advertising industry would like us to think we know,” Steven Tadelis, one of the eBay study’s co-authors, told me.

Ask Google, Facebook, or Twitter, of course, and they’ll reliably bust out third-party research explaining that their ads work just fine, even if consumers don’t always click on them. An entire ecosystem of analytics companies, including big names like ComScore and Nielsen, has evolved to tell clients which online advertisements give them the biggest bang for their bucks. Especially cutting-edge firms, such as Datalogix, have even found ways to draw correlations between the ads consumers see online and what they buy in stores. We are swimming in data. And there are plenty of professionals out there happy to tell corporate America what all that data means, with the help of some fancy mathematical models.

The problem, according to Tadelis and others, is that much of the data websites generate is more or less useless. Some of the problems are practically as old as marketing itself. For instance, companies like to run large ad campaigns during major shopping seasons, like Christmas. But if sales double come December, it’s hard to say whether the ad or the holiday was responsible. Companies also understandably prefer to target audiences they think will like what they’re selling. But that always leads to the nagging question of whether the customer would have gone and purchased the product regardless. Economists call this issue “endogeneity.” Derek Thompson at the Atlantic dubs it the “I-was-gonna-buy-it-anyway problem.”

But the Internet also gunks up attempts at analysis in its own special ways. For instance, if somebody searches for “Amazon, banana slicer,” and clicks on a search ad that pops up right next to his results, chances are he would have made it to Amazon’s site without the extra nudge. Even if he never typed the word Amazon, he still might have gotten to the site through the natural power of search. In the end, it all comes down to the evergreen challenge of distinguishing correlation (e.g., a Facebook user saw an ad and then bought some shoes) from causation (e.g., a Facebook user bought some shoes because he saw an ad).

There is, however, a way to get around these hurdles: Run an experiment. Most analytics companies don’t do that, relying instead on elaborate statistical regressions that try to fix flawed data. But it’s the route Tadelis—now a business professor at the University of California, Berkeley—and his collaborators Tom Blake and Chris Nosko took with eBay. In their first test, the researchers looked at what would happen if the company stopped buying ads next to its own name, which seemed like the most obvious waste of money. To do so, they pulled the ads from Yahoo and MSN but left them running on Google. It turns out, the advertising made virtually no difference. Yet eBay was spending dollars every time a customer clicked an ad instead of the link sitting right below it.



More Than Scottish Pride

Scotland’s referendum isn’t about nationalism. It’s about a system that failed, and a new generation looking to take a chance on itself. 

IOS 8 Comes Out Today. Do Not Put It on Your iPhone 4S.

Why Greenland’s “Dark Snow” Should Worry You

Three Talented Actresses in Three Terrible New Shows

The Human Need to Find Connections in Everything

It’s the source of creativity and delusions. It can harm us more than it helps us.


Happy Constitution Day!

Too bad it’s almost certainly unconstitutional.

The Ungodly Horror of Having a Bug Crawl Into Your Ear and Scratch Away at Your Eardrum

My Father Was James Brown. I Watched Him Beat My Mother. Then I Married Someone Like Him.

  News & Politics
Sept. 17 2014 2:57 PM ISIS Helps Snuff Out Conservative Opposition to Government Funding Bill
Business Insider
Sept. 17 2014 1:36 PM Nate Silver Versus Princeton Professor: Who Has the Right Models?
Sept. 17 2014 1:59 PM Ask a Homo: Secret Ally Codes 
  Double X
The XX Factor
Sept. 17 2014 1:26 PM Hey CBS, Rihanna Is Exactly Who I Want to See on My TV Before NFL Games
  Slate Plus
Slate Fare
Sept. 17 2014 9:37 AM Is Slate Too Liberal?  A members-only open thread.
Brow Beat
Sept. 17 2014 1:01 PM A Rare, Very Unusual Interview With Michael Jackson, Animated
Future Tense
Sept. 17 2014 12:35 PM IOS 8 Comes Out Today. Do Not Put It on Your iPhone 4S.
  Health & Science
Bad Astronomy
Sept. 17 2014 11:18 AM A Bridge Across the Sky
Sports Nut
Sept. 17 2014 3:51 PM NFL Jerk Watch: Roger Goodell How much should you loathe the pro football commissioner?