Faith-based fudging.

The conventional wisdom debunked.
Aug. 5 2003 12:35 PM

Faith-Based Fudging

How a Bush-promoted Christian prison program fakes success by massaging data.

(Continued from Page 1)

The Penn study doesn't conceal the actual poor outcome: All the facts reported above come straight from that report. But the study goes out of its way to put a happy face on the sad results, leading with the graduates-only figures before getting to the grim facts. Apparently, the Prison Fellowship press office simply wrote a press release off the spin, and the White House worked off the press release. Probably no one was actually lying; they were just believing, and repeating as fact, what they wanted to believe. It's hard to know for sure what those involved were thinking: Study author Byron Johnson canceled a scheduled interview at the last moment. The White House didn't respond to requests for comment.

InnerChange program manager Jerry Wilger says he doesn't know much about research, but he doesn't think it's fair to count the performance of the people who dropped out of his program against him, a fair-sounding objection that misses the point entirely. If InnerChange's 177 entrants were truly matched to the control group but ended up having more recidivism, then either the apparent success with the graduates was due to "creaming" or the program somehow managed to make its dropouts worse than they were to start with. If the program genuinely helped its graduates and didn't harm its dropouts, and if the whole group of entrants was truly matched to the controls, then the group of 177 should have done better than the controls. And they didn't.

So, the feel-good winners-only analysis simply isn't worth the paper it's printed on. Only the full-group analysis (known technically as "intent-to-treat," a holdover term from its origins in medical research) has any real value. And on that analysis, the program has a net effect of zero or a little worse than zero. That makes it a loser.

Advertisement

John DiIulio, an intellectually serious advocate of faith-based programs who was the first director of the Bush administration's faith-based initiatives and the founder of the Penn research center, acknowledges frankly the results weren't what a supporter of such programs would have hoped for. But he points out that a single study almost never provides a convincing yes or no answer on a program concept. "The orthodox believers point to a single positive result and say it proves faith-based programs always work. The orthodox secularists point to a single negative result and say it proves faith-based programs never work. They're both wrong."

The poor result of InnerChange doesn't mean that no faith-based prison program could work, but it does mean that this one hasn't, at least not yet. It joins a long line of what seemed like good ideas for reducing recidivism that didn't pan out when subjected to a rigorous evaluation. Maybe my own pet, literacy training, wouldn't do any better in a real random-assignment trial. But that's why you do evaluations; they tell you things you didn't want to hear. If you're honest, you listen to them.

And if you're smart, you don't listen to the political advocates of "faith-based" this and that when they say they're only asking us to support programs that have been "proven" to work.

Correction, Aug. 6, 2003: The article originally and incorrectly described InnerChange Freedom Initiative as "a fundamentalist prison-within-a-prison." The Prison Fellowship regards itself as being part of the evangelical tradition rather than the fundamentalist tradition. (Return to the corrected sentence.)

Mark A.R. Kleiman is a professor of public policy at the University of California–Los Angeles Luskin School of Public Affairs and chairman of BOTEC Analysis, which advised Washington state on implementing its legal cannabis market.

  Slate Plus
Slate Picks
Dec. 19 2014 4:15 PM What Happened at Slate This Week? Staff writer Lily Hay Newman shares what stories intrigued her at the magazine this week.