The statistical games behind "off-label" prescription drug use.

How to fix health policy.
Dec. 21 2009 12:28 PM

Cooking the Books

The statistical games behind "off-label" prescription drug use.

Years ago, mathematician John Allen Paulos described a brilliant stock scam, in which a crooked broker sends out a huge number of letters to potential clients. Half the letters say the market will rise; the other half predict the opposite. As Paulos explains, "No matter whether the index rises or falls, a follow-up letter is sent, but only to [people] who initially received a correct prediction." On that half, the scam is repeated. After several iterations, the broker has hooked some fraction of his initial marks—the guy correctly predicted a half-dozen or so market moves!—and the unsuspecting rubes are ready to be fleeced.

Pharmaceutical companies employed a similar ruse to propel sales of drugs for "off-label" indications, or uses not green-lighted by the Food and Drug Administration. The FDA approves new prescription drugs only after clinical trials show a benefit in the form of a pre-specified outcome, such as fewer asthma attacks, better blood sugar control, or some other metric. (For example, Neurontin was approved to control seizures.) Companies are forbidden to advertise or promote the drug for any other problem. However, no law stops doctors from prescribing the drug for whatever reason they like—and they certainly do.

In 1996, a young researcher named David Franklin left Parke-Davis due to his disgust over off-label marketing of the anti-seizure drug Neurontin. (Franklin reported that one company executive told him, "[W]e need to be … holding [physicians'] hands and whispering in their ear, Neurontin for pain, Neurontin for monotherapy, Neurontin for bipolar, Neurontin for everything.") A resulting class action suit was settled for $430 million in 2004, and thousands of pages of corporate documents soon ended up in a searchable digital library at the University of California-San Francisco.

Three years ago, Stanford researchers found that 20 percent of all prescriptions are written for "off-label" use, such as using an anti-seizure drug to treat attention-deficit disorder, and the vast majority of such uses "had little or no scientific support." By one conservative estimate, the annual U.S. market for off-label prescription drugs is about $44 billion, or about one-fifth of the value of all domestic automobile sales.

According to a remarkable analysis of the Neurontin documents, published last month, many clinical trials of the drug took a shotgun approach. Study patients took the drug, and researchers measured tons of possible outcomes (like pain with touch, pain with cold, excessive pain with pinpricks, more than a dozen different scales for psychiatric symptoms, and so on). By random chance, if you measure enough outcomes, at least some of them will appear better after drug treatment. When the time came to report the findings, however, the researchers systematically omitted the outcomes on which the drug had no effect—and presented only the data showing benefit. That's like dealing dozens of hands of poker to yourself but showing only the hand with good cards.

Like the victims of Paulos' investment swindle, patients and doctors have no way of knowing about the outcomes that weren't winners since they were previously hidden. All they see is that the drug looks like it works for certain off-label indications. In truth, the drug may be no better than a placebo. But rumors build at academic conferences and journals, subtly encouraged by the manufacturers, and doctors start writing prescriptions. At the height of Neurontin's popularity, for example, the drug made $3 billion annually, mostly from patients with problems other than seizures.

What's worrisome is that such statistical manipulation occurs even when no drug companies are clearly involved; an astounding 40 percent of drug trials funded by the Canadian government selectively omitted key outcome data, perhaps because researchers want their pet projects to be winners.

In 2004, in an attempt to fight the problem, the editors of three leading American medical journals decided to publish only clinical trials in which a single outcome was pre-specified and registered with a public database, like clinicaltrials.gov, and many other journals followed this lead. In other words, researchers had to first pick a "primary" outcome like survival from breast cancer, perform the clinical trial of the drug, and then report the primary outcome measures without making excuses. (Notably, the lobbying arm of the drug industry, the Pharmacy Research and Manufacturers of America, didn't endorse public registries.)

But most medical journals are largely supported by pharmaceutical advertising—and many don't take the registration requirements seriously, even though they supposedly required this procedure for all clinical studies. In September, for example, an international group of researchers reviewed clinical trials published in 10 high-impact medical journals in 2008. Fewer than half the published studies were adequately registered. Even among those, about one-third had some unexplained discrepancies in reporting that largely favored the drug under investigation. The review authors concluded, depressingly, that "selective outcome reporting is prevalent."

To be sure, some off-label uses are important, particularly in rare conditions without accepted treatments, such as unusual cancers or rare genetic diseases. But for the most part, the proliferation of off-label prescription undercuts legitimate medical practice since it skirts accepted safety and efficacy standards. (Most often, psychiatric drugs are the offender. An example: Doctors increasingly prescribe the anti-hyperactivity drug Adderal as a weight loss aide for overweight kids.) Unfortunately, the FDA recently loosened restrictions on manufacturers' ability to promote off-label use—companies are now allowed to distribute scientific papers promoting off-label use, within certain boundaries.

Without greater oversight from medical journals, the public must rely on whistle-blowers to come forward, as with Neurontin (such folks collect a hefty bounty from damage awards). But it would certainly be better for our collective health if clinical studies were kept honest from the beginning.

Become a fan of Slate on Facebook. Follow us on Twitter.

TODAY IN SLATE

Politics

Don’t Worry, Obama Isn’t Sending U.S. Troops to Fight ISIS

But the next president might. 

IOS 8 Comes Out Today. Do Not Put It on Your iPhone 4S.

Why Greenland’s “Dark Snow” Should Worry You

How Much Should You Loathe NFL Commissioner Roger Goodell?

Here are the facts.

Three Talented Actresses in Three Terrible New Shows

Science

The Human Need to Find Connections in Everything

It’s the source of creativity and delusions. It can harm us more than it helps us.

Foreigners

More Than Scottish Pride

Scotland’s referendum isn’t about nationalism. It’s about a system that failed, and a new generation looking to take a chance on itself. 

The Ungodly Horror of Having a Bug Crawl Into Your Ear and Scratch Away at Your Eardrum

We Could Fix Climate Change for Free. Now There’s Just One Thing Holding Us Back.

  News & Politics
Weigel
Sept. 17 2014 7:03 PM Once Again, a Climate Policy Hearing Descends Into Absurdity
  Business
Business Insider
Sept. 17 2014 1:36 PM Nate Silver Versus Princeton Professor: Who Has the Right Models?
  Life
Outward
Sept. 17 2014 6:53 PM LGBTQ Luminaries Honored With MacArthur “Genius” Fellowships
  Double X
The XX Factor
Sept. 17 2014 6:14 PM Today in Gender Gaps: Biking
  Slate Plus
Slate Fare
Sept. 17 2014 9:37 AM Is Slate Too Liberal?  A members-only open thread.
  Arts
Brow Beat
Sept. 17 2014 5:56 PM Watch Louis C.K., Dave Chappelle, Bill Hicks, Mitch Hedberg, and More on New YouTube Channel
  Technology
Future Tense
Sept. 17 2014 7:23 PM MIT Researchers Are Using Smartphones to Interact With Other Screens
  Health & Science
Jurisprudence
Sept. 17 2014 4:49 PM Schooling the Supreme Court on Rap Music Is it art or a true threat of violence?
  Sports
Sports Nut
Sept. 15 2014 9:05 PM Giving Up on Goodell How the NFL lost the trust of its most loyal reporters.