How to Know Which Pregnancy Studies Are Legit

Pregnancy by the Numbers
Aug. 28 2013 7:14 AM

How to Know Which Pregnancy Studies Are Legit

woman_frustrated
Familiar?

Photo by wavebreakmedia/Shutterstock

Most weeks, CNN Health or the New York Times Science section (or Slate!) reports on another study about health. Within the past couple weeks there was one about how four cups of coffee a day kills you and one on which brands of adult beverages are most likely to result in a trip to the emergency room (answer: malt liquor).

Emily Oster Emily Oster

Emily Oster is an associate professor of economics at the University of Chicago Booth School. Her book is Expecting Better: Why the Conventional Pregnancy Wisdom Is Wrong. Follow her on Twitter.

A question that comes up again and again in reading these, and came up all the time when I was writing my book on pregnancy, is how to know which studies warrant our attention.

Advertisement

The gold standard in medicine, and in other fields, is the randomized controlled trial. In a study like this, participants are divided into two (or more) groups randomly, and each is told to do something different. In a drug study, one group takes a drug and the other does not. Because the groups are randomly selected, on average they are similar before the study. So if the researchers see differences after the study, they can be confident the differences are due to the treatment.

Even in a randomized study, there are limitations. No research study is run on the entire population of the world. What we learn from these studies is the impact of the treatment on average, not on every individual.

So if you are approaching a health decision based on data from a randomized study, that’s great. But the majority of the time, especially in public health where I looked for data on pregnancy, the studies aren’t randomized. In 2012, the American Journal of Public Health published 128 papers—only 14 of them were randomized.  

What we get instead are observational studies, which also compare individuals who engage in different behaviors. The difference, though, is that these individuals choose to engage in these behaviors on their own—and if differences among people influence their choice of behavior, it may be those differences, not the behavior itself, that changes their outcomes.

Let’s consider a particular example, drawn from pregnancy (since this is, after all, a pregnancy blog). When I came to look at the issue of caffeine, I found that, at least in evaluating the relationship with miscarriage, there were no randomized studies. So I turned to observational studies.

One that I summarize in my book is from the American Journal of Obstetrics and Gynecology, published in 2008. The paper compares women who drink no caffeine to those who drink two cups a day or more and analyzes the risk of miscarriage (the researchers also included an intermediate group, but for now we’ll think about the simple comparison). Table 1 of this paper summarizes the characteristics of women in these different caffeine groups. Here’s a subset:

oster_graph

Clearly, there are big differences across these groups other than how much caffeine they drink. The coffee drinkers are older, more likely to be white, poorer on average, and more likely to smoke. Smoking and age, in particular, are linked to miscarriage. So is it the coffee? Or is it the smoking?

The crucial element of evaluating these studies is to figure out how big these differences really are and how much they matter. It’s common in studies like this to show the impacts of the treatment—in this case, caffeine—after adjusting for these controls. This adjustment is important but also incomplete. In the above study, for instance, the authors controlled for whether the household income was more or less than $50,000, but that’s only a crude measure. It is hard to know whether better controls—more detail about income— would make more of a difference.  

What does this mean, in general, when we’re faced with these studies? How can we evaluate them? One thing to start with is always be a little skeptical. But a bit more concretely, here are two tips:

1. Look at how different the groups are on factors such as age, education, and so on. All else equal, the more similar the groups look, the better.

2. Look at how complete the data is. A lot of media reports say a study is “adjusted for socio-demographics.” How effective this is depends on how comprehensive the variables on demographics are. A study using data that details exactly how much education someone has, and exactly their income will be able to adjust more completely for this issue than one where all the researchers observe is whether someone completed high school or not.

If you use these criteria, you’ll quickly realize that some studies are much, much better than others. And while no single study is going to be enough to close the book on any issue, thinking critically about these possible problems may sometimes lead you to decide a study doesn’t have much information at all. Not to mention allow you to drink coffee.

TODAY IN SLATE

Politics

Don’t Worry, Obama Isn’t Sending U.S. Troops to Fight ISIS

But the next president might. 

The Extraordinary Amicus Brief That Attempts to Explain the Wu-Tang Clan to the Supreme Court Justices

Amazon Is Officially a Gadget Company. Here Are Its Six New Devices.

The Human Need to Find Connections in Everything

It’s the source of creativity and delusions. It can harm us more than it helps us.

How Much Should You Loathe NFL Commissioner Roger Goodell?

Here are the facts.

Altered State

The Plight of the Pre-Legalization Marijuana Offender

What should happen to weed users and dealers busted before the stuff was legal?

Surprise! The Women Hired to Fix the NFL Think the NFL Is Just Great.

You Shouldn’t Spank Anyone but Your Consensual Sex Partner

Moneybox
Sept. 17 2014 5:10 PM The Most Awkward Scenario in Which a Man Can Hold a Door for a Woman
  News & Politics
Altered State
Sept. 17 2014 11:51 PM The Plight of the Pre-Legalization Marijuana Offender What should happen to weed users and dealers busted before the stuff was legal?
  Business
Business Insider
Sept. 17 2014 1:36 PM Nate Silver Versus Princeton Professor: Who Has the Right Models?
  Life
Outward
Sept. 17 2014 6:53 PM LGBTQ Luminaries Honored With MacArthur “Genius” Fellowships
  Double X
The XX Factor
Sept. 17 2014 6:14 PM Today in Gender Gaps: Biking
  Slate Plus
Slate Fare
Sept. 17 2014 9:37 AM Is Slate Too Liberal?  A members-only open thread.
  Arts
Brow Beat
Sept. 17 2014 8:25 PM A New Song and Music Video From Angel Olsen, Indie’s Next Big Thing
  Technology
Future Tense
Sept. 17 2014 9:00 PM Amazon Is Now a Gadget Company
  Health & Science
Medical Examiner
Sept. 17 2014 11:48 PM Spanking Is Great for Sex Which is why it’s grotesque for parenting.
  Sports
Sports Nut
Sept. 17 2014 3:51 PM NFL Jerk Watch: Roger Goodell How much should you loathe the pro football commissioner?