Science

Bad Footnotes Can Be Deadly

A one-paragraph blurb helped cause the opioid crisis. That’s just the start of science’s citation woes.

39351701oxyc_20020322_00366.jpg
The number of citations of the blurb spiked in 1996, researchers note, just after OxyContin was brought to market.

Darren McCollester/Getty Images

The New York Times reported last week that 59,000 people died from drug overdoses in 2016, in the latest sign that America’s prescription painkiller epidemic is only getting worse. Yet the more shocking news about the scourge of opioids came a few days earlier, in a note published in the New England Journal of Medicine by a team of researchers in Canada. That note shows how a tiny blurb that first appeared in the journal’s January 1980 issue helped reshape—and distort—conventional wisdom on pain management, tilting doctors in favor of giving out addictive drugs.

Back in 1979, Boston University Medical Center researchers Jane Porter and Hershel Jick found that just a handful of the patients who’d been treated with narcotics at a set of six hospitals went on to develop drug dependencies. Their single-paragraph summary of this result would be published as a letter to the editor in the NEJM under the heading, “Addiction Rare in Patients Treated With Narcotics.”

According to the recent correspondence in NEJM, this single paragraph was cited hundreds of times in the 1990s and 2000s to support the claim that prescription painkillers weren’t that addictive. It was during this period that doctors started treating pain much more aggressively than they had before and handing out potent drugs with little circumspection. (For a good history of the changing use of painkillers, see this piece in Vox.)

The original paragraph from Porter and Jick, just 101 words in all, read as follows:

Recently, we examined our current files to determine the incidence of narcotic addiction in 39,946 hospitalized medical patients who were monitored consecutively. Although there were 11,882 patients who received at least one narcotic preparation, there were only four cases of reasonably well documented addiction in patients who had no history of addiction. The addiction was considered major in only one instance. The drugs implicated were meperidine in two patients, Percodan in one, and hydromorphone in one. We conclude that despite widespread use of narcotic drugs in hospitals, the development of addiction is rare in medical patients with no history of addiction.

Most citations of this note seemed to overlook its narrow scope. The blurb gives a rate of drug addiction for patients with restricted access to the drugs in question and no stated definition of what it means to be “addicted.” Thirty-seven years ago, when Porter and Jick’s letter first appeared, opioids were carefully controlled; the patients they described may have taken painkillers while they were at the hospital, but they weren’t going home with them. That meant the addiction rate they found (four in 11,882) had little bearing on the more important question—now sadly resolved—of whether it’s safe to prescribe opioids to patients outside the hospital setting. Despite these limitations, the stature of this tiny research project seemed to only grow as time went on, like a scholarly fish tale. In 1990, Scientific American described Porter and Jick’s paragraph as a “an extensive study.” By 2001, Time had promoted it to the status of “a landmark study.”

The new NEJM paper rightly notes “the need for diligence when citing previously published studies.” Yet the Canadian authors were not the first to write on how the Porter and Jick reference spread in altered form and influenced the field. Journalist Sam Quinones, for one, traced the history of the 1980 paragraph in great detail in his award-winning 2015 book on the opioid crisis, Dreamland. (The above references to Scientific American and Time come from him.) His careful work on the topic isn’t mentioned anywhere in the recent publication. Nor is Pain Killer, a book by the New York Times’ Barry Meier that called attention to the suspect use of Porter and Jick way back in 2003. While highlighting the sloppiness of citation practice, and its sometimes deadly consequences,  the Canadians themselves failed to cite some worthy references. (The paper’s senior author, David Juurlink, did give credit to Meier and Quinones in other settings.)

This is perhaps a minor transgression, but it gets at a larger point. The case of Porter and Jick has been treated as an aberration in the research literature, with the Canadians even hinting that the sloppy references were in part a ploy by pharmaceutical executives. (The number of citations spiked in 1996, they note, just after OxyContin was brought to market.) But this pattern of mistakes need not represent some special case of bibliographic malfeasance. Rather, it’s a standard feature of the research literature. Our system of citations is so ridden with inaccuracy that it’s sometimes useless.

In academic publishing, references are meant to buttress arguments, establish facts, and dole out credit where it’s due. In practice, they often do the opposite, hiding more than they show. In a disturbing and delightful series of papers on this topic, Norwegian social anthropologist Ole Bjorn Rekdal has shown how easily and often citations are abused. When you try to trace the provenance of any given, referenced fact—on addiction rates, for example—you may well find yourself tangled in a nest of secondary sources, with each paper claiming to have pulled the fact from another. These daisy-chained citations make it very hard—and at times impossible—to locate original source material. They also lead to a game of research telephone, in which the context of a fact gets stripped away, and its meaning morphed as it gets transmitted from one citation to the next.

That’s how the Porter and Jick paper came to grow in stature from a single paragraph to an “extensive study” and then a “landmark” one. But to make things worse, scholars often hide the game of telephone: Instead of citing facts to whatever secondary sources they consulted, they’ll sometimes cite the original while never having checked it for themselves. (Rekdal calls this “citation plagiarism.”) The problem there is that whatever distortions have been introduced along the way will then be referenced, inaccurately, straight back to the primary source, thus giving them more credence. This habit gives rise to “academic urban legends”—unfounded rumors, errors, or exaggerations that reproduce themselves in the scholarly literature. It’s the researcher’s equivalent of saying, This crazy thing happened to my friend, when in fact her story may have been filtered through many layers of friends of friends.

One can see this happening in the references to Porter and Jick. Their original paragraph hasn’t just been cited in misleading ways. It’s been cited in consistently misleading ways, as if the distorting glosses on their research had been carbon-copied. Rekdal notes that among the earliest, most influential papers citing the 1980 paragraph was a 1990 expert report on palliative care from the World Health Organization. That paper’s reference includes a telling glitch: Instead of giving the title of Porter and Jick’s paragraph as “Addiction Rare in Patients Treated With Narcotics,” the WHO paper swaps out an r for a t in the second word, citing the study as “Addiction Rate in Patients Treated With Narcotics.”

This typo was surely inadvertent, but it reframed the reference in a subtle way. “The word rate suggests that there were precise calculations based on scientific methods,” says Rekdal, “strengthening the misconception that Porter and Jick was a scientific article, and not a five-sentence letter to the editor providing little or no context or evidence to the conclusion in the title.” Sure enough, that typo has been repeated ever since, in scientific papers, evidence reviews, patient websites, and even textbooks on pharmacology and neurology.

Why would so many sources have cited the Porter and Jick paragraph in the same, erroneous fashion? To Rekdal, the most likely explanation is that these subsequent authors borrowed both the interpretation and the reference from the WHO paper, or some other secondary source, without ever reading the original text. “Many of the authors who cited Porter and Jick probably did not even know that they cited a one-paragraph letter to the editor, and not a sound scientific article,” he says.

Again, this sort of thing happens all the time. Rekdal has collected many examples of academic urban legends. Among his favorites is a famous line about the importance of making accurate citations, which first appeared in an article by Katherine Frost Bruner in 1942: “A sin one more degree heinous than an incomplete reference,” she wrote, “is an inaccurate reference; the former will be caught by the editor or the printer, whereas the latter will stand in print as an annoyance to future investigators and a monument to the writer’s carelessness.” This quote, or at least a portion of it, would show up in several editions of a ubiquitous academic style guide, the Publication Manual of the American Psychological Association, as well as many other places. According to Rekdal, it’s now appeared in books, articles, and websites in nine different languages, always in the context of providing guidelines or advice on how best to write and source an academic text.

The problem is, the most well-known version of this quote—the one that’s been repeated in the literature—mixes up the words in a way that distorts its meaning. The third edition of the American Psychological Association’s Publication Manual, published in 1984, summarizes Bruner’s article to say that both incomplete and inaccurate references (as opposed to just the latter) would serve as “monuments to the writer’s carelessness.” In other words, a citation error—a monument to carelessness, if you will—appears even in the authoritative text on the proper, accurate use of citations. That mistake was in turn repeated in the fourth and fifth editions of the manual, and has since been reproduced and passed along in many other sources.

While the academic literature is teeming with stories like this, it takes extraordinary effort—and the arduous navigation of Matryoshka-doll sourcing—to get to the bottom of any one of them. An ecologist named James Wetterer, for example, decided to look into an often-told story of invasive ants on the islands of Madeira. According to numerous citations in the literature, careful work from the 1800s had established two successive waves of colonization by foreign ants and the corresponding extinctions of the local species. To Wetterer, this seemed far-fetched: His own, 21st-century work had found that native species were still thriving on the islands and that invasive species never occupied more than 10 percent of the archipelago. To figure out how the other story might have spread, he consulted 55 secondary sources and managed to trace the false story of the Madeira ant extinctions to a group of papers that came out in the early 1900s. No one ever cited those mistaken papers, though. They just repeated their mistake while pretending to have consulted older research.

“Most scientists probably have some awareness [of] quotation error and citation copying in the scientific literature,” Wetterer wrote, “but I believe few have much appreciation for how common or important these problems may be.” He goes on to summarize some broader surveys of the problem: One study compared more than 1,000 direct quotes in scholarly papers with their original sources and found that 44 percent contained at least one mistake; another looked at how mistakes like typos propagate through bibliographies and concluded that at least 70 percent of all scientific citations are copied from the bibliographies of other secondary sources.

Other researchers have found error rates as high as 67 percent in the journals of specific fields. Rekdal notes that entire books are routinely cited as the source for specific facts, without the help of page numbers. “At times, I get the feeling that references have been placed in quantities and with a degree of precision reminiscent of last minute oregano flakes being sprinkled over a pizza on the way to the oven,” he writes.

The weirdest thing about this problem is that technology has done so little to arrest it. Consider the case we started with, of Porter and Jick (1980). As Quinones points out, until 2010 the New England Journal of Medicine did not have online archives going back before 1993. That meant any researcher who wanted to verify the contents of the original reference would have had to go to the library and pull up a paper copy. Now it’s possible to find that infamous paragraph—and the list of papers citing it—with just a few clicks. Given this development, it should be easier than ever to smash the old monuments to carelessness and avoid erecting new ones.

In practice, though, the spread of information online has only made things worse. Increased pressure on scholars to pad their résumés—and the proliferation of outlets for their work—may have helped prioritize quantity over quality in academic publishing. There simply isn’t time to double-check every reference, even with the digital tools that would make that process so much more efficient than it’s ever been before.

Meanwhile, the same digital tools that might be used to clean up the literature also make it easier to rank scholars according to their “bibliometrics”—i.e., the number of times their papers have been cited in the literature. This, in turns incentivizes researchers to use (and abuse) their bibliographies as a way of advancing their careers. I mean, why not stuff your paper full of vacuous pointers to your own work or that of your colleagues?

The stuffing of journals with trash citations has clogged a vital channel of scientific communication, by overwhelming useful references with those that are trivial, inaccurate, or plagiarized. The recent flap over Porter and Jick’s paragraph from 1980 shows how this knowledge jamming can even, in some cases, be a matter of life or death.