Cooking the School Books

Cooking the School Books

Cooking the School Books

Dumb ideas exposed here.
Sept. 1 1999 3:30 AM

Cooking the School Books

How U.S. News cheats in picking its "best American colleges."


Click here for a response to this article by the editors of U.S. News.


According to the annual "America's Best Colleges" issue of U.S. News & World Report, published Aug. 30, the best college in the United States is the California Institute of Technology. This was dramatic, since Caltech, while highly regarded, is not normally thought of as No. 1. Last year Caltech was rated No. 9, while the top spot was an uninteresting three-way tie among Harvard, Yale, and Princeton.

"Why does U.S. News rank colleges?" asks U.S. News. The "simple answer," the magazine says, is, "We do it to help you make one of the most important decisions of your life." Perhaps. Another simple answer is that the annual college rankings (and similar rankings of graduate schools and hospitals) are lucrative and influential unlike anything else the No. 3 newsmag does. Newsstand sales are almost double those of a normal issue, and a paperback-book version sells a million copies. Colleges brag or complain loudly about their scores, enhancing the 'Snooze either way.

Whatever their validity as measures of academic excellence, the annual rankings are a brilliant gimmick for U.S. News. But there's a problem. A successful feature like this requires surprise, which means volatility. Nobody's going to pay much attention if it's Harvard, Yale, and Princeton again and again, year after year. Yet the relative merits of America's top universities surely change slowly, if at all. Naturally, U.S. News does not just make up its ratings. It uses a weighted average of 16 numerical factors such as average class size, acceptance rate (fraction of applicants who are admitted), and amount of alumni giving. Trouble is, any combination of these factors just isn't going to change enough from year to year to keep things interesting.

So how on earth can U.S. News explain Caltech's one-year rise?

The magazine tries to deny that there's anything odd about a college improving so quickly. The "best colleges" story argues: "Caltech has always been within striking distance of the top of the chart. In 1989, Caltech was the No. 3 school, ahead of Harvard. ... Last year, Caltech had the fourth-highest score among national universities." The first assertion is irrelevant: We're not interested in the 10-year rise from third but rather in the one-year rise from ninth. The second assertion is technically true but practically dishonest: Caltech had the "fourth-highest score" last year only because there were two three-way ties and one two-way tie among the eight schools that beat it.

But the real reason Caltech jumped eight spaces this year is that the editors at U.S. News fiddled with the rules. The lead story of the "best colleges" package says that a change in "methodology ... helped" make Caltech No. 1. Buried in a sidebar is the flat-out concession that "[t]he effect of the change ... was to move [Caltech] into first place." No "helped" about it. In other words, Caltech didn't improve this year, and Harvard, Yale, and Princeton didn't get any worse. If the rules hadn't changed, HYP would still be ahead. If the rules had changed last year, Caltech would have been on top a year earlier.

(In fact, if the U.S. News criteria are taken seriously, and if they held steady, Caltech may actually have slipped in quality this past year. Most indicators did not change compared to last year. But graduation rate, number of classes with fewer than 20 students, and percentage of faculty members who work full-time actually declined. Only two indicators showed small improvements: percentage of accepted students in top 10 percent of their high-school classes went from 99 percent to 100 percent [big deal!], and Caltech's acceptance rate fell from 23 percent to 18 percent.)

U.S. News denies that it changes the rules--as it does every year--simply to change the results. Robert Morse, U.S. News' statistical guru, explained to me that this year's ranking procedures are an "improvement" over last year's. Doesn't that imply, I said, that last year's rankings were inferior? And shouldn't U.S. News apologize to anyone who made "one of the most important decisions of your life"--possibly turning down Caltech for Princeton--based on rankings the magazine itself now regards as inaccurate? Morse replied that he hadn't said the earlier ratings were inferior. But if something improves, I pressed him, doesn't that mean that it was less excellent before the improvement? Morse grudgingly allowed that I was free to make that inference.

I can't prove that U.S. News keeps changing the rules simply in order to change the results. But if not, U.S. News ought to shy away from horse-race headlines such as "Caltech Comes out on Top." A more honest summary might be "We Finally Realize That Caltech Is Tops." Or "Caltech on Top (Until We Fiddle With Rules Again)."