Science undervalues novelty. How can we encourage more iconoclasm?

Pathbreaking Study Shows Researchers Don’t Publish Enough Pathbreaking Studies

Pathbreaking Study Shows Researchers Don’t Publish Enough Pathbreaking Studies

The search for better economic policy.
June 28 2016 9:47 AM

We Need More Iconoclasts!

Science undervalues novelty. How can we encourage researchers to take greater risks?

We need more iconoclastic researchers.

Stockbyte/Thinkstock

Economist George Akerlof changed the way economists practice their craft, laid the intellectual foundation for work that has nurtured the development of online markets, and won a Nobel Prize for his efforts. He is one of the towering intellects of modern economics.

We offer this glowing description with the benefit of hindsight. In the decades since Akerlof’s paper “The Market for ‘Lemons’ ” was published, it’s become one of the most frequently cited papers in the social sciences. But in the 1960s, Akerlof was just another young assistant professor trying to get his rather unorthodox paper on the economics of the used car market published in an academic journal. He had developed a spare, elegant model of what happens when one side of a market knows more than the other, using used car sales as a way of explaining his model’s stark predictions: When there is enough of a gap between what the two sides know—if you can’t tell a good used car (a cherry) from a bad one (a lemon)—better products are taken off the market by their owners, and the quality of available goods spirals downward until there’s nothing for sale at all and the market collapses.

Advertisement

At the time, Akerlof’s approach of reality-based modeling, focused on a single illustrative example, confounded the profession’s old guard. His work came at a moment when the aspirations of mainstream economists ran to ever-greater generality with the ultimate goal of capturing the salient features of an entire economy in mathematical abstraction: This was a generation of economists who’d been trained to think like physicists. Akerlof discovered this soon after submitting his work to a leading academic journal, only to be told by the editor that they “did not publish papers on subjects of such triviality.” Others thought his example was self-evident or outright wrong. (As one reviewer noted, “I can see a market for used cars just outside my window.”)

It’s a classic story of the iconoclast who is rejected by the establishment only to be vindicated with the passage of time. Of course, there are also plenty of counterexamples—iconoclasts who were just plain wrong.

So are iconoclasts good or bad for progress? In recent years, researchers have turned their sights on themselves, exploring whether radical ideas tend to be revolutionary or duds. As you might have guessed, they are both. But they are groundbreaking often enough that the figure of the underappreciated iconoclast is very real. And we’d be doing science and progress a favor if we’d show such pathbreakers a little more love.

A pair of researchers from Belgium, Jian Wang and Reinhilde Veugelers, teamed up with Georgia State economist Paula Stephan to provide the most recent contribution to the growing evidence of the value of unconventional ideas. They measure the novelty of a scientific paper based on the extent to which it draws on prior work from journals that have never before been cited together, and give added weight to papers that draw on sources from very different disciplines. For example, it is very conventional for a paper on cells to cite both Cell and the Journal of Cell Biology. A paper would be coded by the researchers as more novel if it built on cited work from Cell together with, say, the Journal of Materials Engineering. (This measure will, by definition, miss innovations that don’t involve unusual combinations of prior knowledge.) Because it may take a while for more unusual work to be appreciated, the authors look at how often a paper gets cited in the 14 years following publication. (Usual measures of impact consider a much shorter window.)

Advertisement

With their measure of novelty in hand, Stephan et al. show that novel research generates more citations but also has much higher variability in outcomes: Novel work leads to many more strikeouts—low-cited flops—than conventional research, but also to more home runs. They also find that more time is required for unconventional work to gain acceptance: Initially, papers that hold to standard conventions are more likely to be hits in the near term but are surpassed within a few years by more novel work. These findings echo those of an earlier study conducted, appropriately enough, by an interdisciplinary group including an economist, a sociologist, and two physicists, who found that research that is mostly grounded in one discipline, but draws on some novel ideas from outside its main area, is most apt to be high impact.

Another article by Akerlof illustrates the point.* His paper “Labor Contracts as Partial Gift Exchange,” first published in 1982, is focused on understanding why companies may voluntarily pay above-market wages. It’s mostly a labor economics paper and primarily cites prior work by labor economists. But it also sprinkles in references to psychologists and sociologists to introduce the idea of reciprocity into the worker-firm relationship—well before “behavioral economics” was all the rage. Akelof raised the intriguing possibility that companies might pay higher-than-necessary wages because grateful employees would return the favor by working extra hard as a result. It is now literally required reading for Ph.D. students in labor economics, but it took quite a number of years to gain acceptance: It’s received more than four times as many citations in the past half-dozen years as it did in the decade after it was published.

The benefits of iconoclasm wouldn’t be a problem if scientific funders and journal referees were willing to take more chances on unusual work, recognizing that there might be a payoff 10 or so years down the road. But Stephan et al also find that top journals are impatient: novel work tends to get published in less prestigious, lower-impact journals. Who can blame journal editors? After all, standard measures of influence like impact factor, which measures the frequency with which the average article in a journal is cited in a particular year, tend to focus on the influence of only recently published papers.

The need for a quick payback afflicts funding decisions as well. The National Institutes of Health, which distributes tens of billions of dollars in research funding annually, supports specific projects or experiments over just a few years, and requires favorable initial results before providing funding. This all but ensures that NIH-funded research moves science forward by baby steps.

Advertisement

The bias in publication toward the conventional, combined with the way research is funded, adds up to a system that rewards incremental advances over breakthroughs. A study published a few years ago provides a sense of how much these rewards matter by comparing NIH-funded researchers to scientists who are supported by the Howard Hughes Medical Institute, which gives its award-winners flexible support that lasts a full 10 years. Howard Hughes scholars use their longtime horizon to take bigger risks, generating twice as many blockbuster papers—those in the top 1 percent of the citation distribution—as NIH-funded researchers. (A more detailed description of this study appeared in Slate in 2010.)

To be clear, we don’t mean to suggest that the current system has been thoughtlessly setup to curtail scientific progress. There’s a reason why the federal government wants to know whether NIH research dollars are bearing fruit in the short run: NIH-funded work might not be producing meaningful research because it requires more time to realize its potential, or because the work is just no good. It’s easier for the NIH to be cautious than to stand up for work that’s as of yet not bearing fruit. As for researchers deciding where to submit their work, it’s useful to have an indication of where the best research is appearing right now. You don’t care whether the journal had a brilliant editor 15 years ago.

We would argue, however, that all extant evidence indicates, at a minimum, that some tweaks to the current system could be a boon to scientific progress. We could think bigger, think long-term, and most important, recognize that along the way there will be washouts and failures and not give up when they inevitably arise—it’s all part of the process of scientific exploration. George Akerlof’s paper on asymmetric information and the used car market—as trivial and novel as it first appeared to journal editors—has led in turn to many more advances in how we think about information asymmetries in markets for everything from health care to bank loans. Without more breakthroughs of this sort, even accounting for the inevitable failures, the world will be a poorer place.

Correction, July 1, 2016: The article originally stated that the 1982 paper "Labor Contracts as Partial Gift Exchange" was co-authored by George Akerlof and his wife, Janet Yellen. Akerlof wrote the paper solo. (Return.)