Cooking the School Books

Dumb ideas exposed here.
Sept. 1 1999 3:30 AM

Cooking the School Books

How U.S. News cheats in picking its "best American colleges."


Click here for a response to this article by the editors of U.S. News.


According to the annual "America's Best Colleges" issue of U.S. News & World Report, published Aug. 30, the best college in the United States is the California Institute of Technology. This was dramatic, since Caltech, while highly regarded, is not normally thought of as No. 1. Last year Caltech was rated No. 9, while the top spot was an uninteresting three-way tie among Harvard, Yale, and Princeton.

"Why does U.S. News rank colleges?" asks U.S. News. The "simple answer," the magazine says, is, "We do it to help you make one of the most important decisions of your life." Perhaps. Another simple answer is that the annual college rankings (and similar rankings of graduate schools and hospitals) are lucrative and influential unlike anything else the No. 3 newsmag does. Newsstand sales are almost double those of a normal issue, and a paperback-book version sells a million copies. Colleges brag or complain loudly about their scores, enhancing the 'Snooze either way.

Whatever their validity as measures of academic excellence, the annual rankings are a brilliant gimmick for U.S. News. But there's a problem. A successful feature like this requires surprise, which means volatility. Nobody's going to pay much attention if it's Harvard, Yale, and Princeton again and again, year after year. Yet the relative merits of America's top universities surely change slowly, if at all. Naturally, U.S. News does not just make up its ratings. It uses a weighted average of 16 numerical factors such as average class size, acceptance rate (fraction of applicants who are admitted), and amount of alumni giving. Trouble is, any combination of these factors just isn't going to change enough from year to year to keep things interesting.

So how on earth can U.S. News explain Caltech's one-year rise?

The magazine tries to deny that there's anything odd about a college improving so quickly. The "best colleges" story argues: "Caltech has always been within striking distance of the top of the chart. In 1989, Caltech was the No. 3 school, ahead of Harvard. ... Last year, Caltech had the fourth-highest score among national universities." The first assertion is irrelevant: We're not interested in the 10-year rise from third but rather in the one-year rise from ninth. The second assertion is technically true but practically dishonest: Caltech had the "fourth-highest score" last year only because there were two three-way ties and one two-way tie among the eight schools that beat it.

But the real reason Caltech jumped eight spaces this year is that the editors at U.S. News fiddled with the rules. The lead story of the "best colleges" package says that a change in "methodology ... helped" make Caltech No. 1. Buried in a sidebar is the flat-out concession that "[t]he effect of the change ... was to move [Caltech] into first place." No "helped" about it. In other words, Caltech didn't improve this year, and Harvard, Yale, and Princeton didn't get any worse. If the rules hadn't changed, HYP would still be ahead. If the rules had changed last year, Caltech would have been on top a year earlier.

(In fact, if the U.S. News criteria are taken seriously, and if they held steady, Caltech may actually have slipped in quality this past year. Most indicators did not change compared to last year. But graduation rate, number of classes with fewer than 20 students, and percentage of faculty members who work full-time actually declined. Only two indicators showed small improvements: percentage of accepted students in top 10 percent of their high-school classes went from 99 percent to 100 percent [big deal!], and Caltech's acceptance rate fell from 23 percent to 18 percent.)

U.S. News denies that it changes the rules--as it does every year--simply to change the results. Robert Morse, U.S. News' statistical guru, explained to me that this year's ranking procedures are an "improvement" over last year's. Doesn't that imply, I said, that last year's rankings were inferior? And shouldn't U.S. News apologize to anyone who made "one of the most important decisions of your life"--possibly turning down Caltech for Princeton--based on rankings the magazine itself now regards as inaccurate? Morse replied that he hadn't said the earlier ratings were inferior. But if something improves, I pressed him, doesn't that mean that it was less excellent before the improvement? Morse grudgingly allowed that I was free to make that inference.

I can't prove that U.S. News keeps changing the rules simply in order to change the results. But if not, U.S. News ought to shy away from horse-race headlines such as "Caltech Comes out on Top." A more honest summary might be "We Finally Realize That Caltech Is Tops." Or "Caltech on Top (Until We Fiddle With Rules Again)."



Blacks Don’t Have a Corporal Punishment Problem

Americans do. But when blacks exhibit the same behaviors as others, it becomes part of a greater black pathology. 

I Bought the Huge iPhone. I’m Already Thinking of Returning It.

Scotland Is Just the Beginning. Expect More Political Earthquakes in Europe.

Lifetime Didn’t Think the Steubenville Rape Case Was Dramatic Enough

So they added a little self-immolation.

Two Damn Good, Very Different Movies About Soldiers Returning From War

Medical Examiner

The Most Terrifying Thing About Ebola 

The disease threatens humanity by preying on humanity.

Students Aren’t Going to College Football Games as Much Anymore, and Schools Are Getting Worried

The Good Wife Is Cynical, Thrilling, and Grown-Up. It’s Also TV’s Best Drama.

  News & Politics
Sept. 20 2014 11:13 AM -30-
Business Insider
Sept. 20 2014 6:30 AM The Man Making Bill Gates Richer
Sept. 20 2014 7:27 AM How Do Plants Grow Aboard the International Space Station?
  Double X
The XX Factor
Sept. 19 2014 4:58 PM Steubenville Gets the Lifetime Treatment (And a Cheerleader Erupts Into Flames)
  Slate Plus
Slate Picks
Sept. 19 2014 12:00 PM What Happened at Slate This Week? The Slatest editor tells us to read well-informed skepticism, media criticism, and more.
Brow Beat
Sept. 20 2014 3:21 PM “The More You Know (About Black People)” Uses Very Funny PSAs to Condemn Black Stereotypes
Future Tense
Sept. 19 2014 6:31 PM The One Big Problem With the Enormous New iPhone
  Health & Science
Bad Astronomy
Sept. 20 2014 7:00 AM The Shaggy Sun
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.