Cooking the School Books

Dumb ideas exposed here.
Sept. 1 1999 3:30 AM

Cooking the School Books

How U.S. News cheats in picking its "best American colleges."


Click here for a response to this article by the editors of U.S. News.


According to the annual "America's Best Colleges" issue of U.S. News & World Report, published Aug. 30, the best college in the United States is the California Institute of Technology. This was dramatic, since Caltech, while highly regarded, is not normally thought of as No. 1. Last year Caltech was rated No. 9, while the top spot was an uninteresting three-way tie among Harvard, Yale, and Princeton.

"Why does U.S. News rank colleges?" asks U.S. News. The "simple answer," the magazine says, is, "We do it to help you make one of the most important decisions of your life." Perhaps. Another simple answer is that the annual college rankings (and similar rankings of graduate schools and hospitals) are lucrative and influential unlike anything else the No. 3 newsmag does. Newsstand sales are almost double those of a normal issue, and a paperback-book version sells a million copies. Colleges brag or complain loudly about their scores, enhancing the 'Snooze either way.

Whatever their validity as measures of academic excellence, the annual rankings are a brilliant gimmick for U.S. News. But there's a problem. A successful feature like this requires surprise, which means volatility. Nobody's going to pay much attention if it's Harvard, Yale, and Princeton again and again, year after year. Yet the relative merits of America's top universities surely change slowly, if at all. Naturally, U.S. News does not just make up its ratings. It uses a weighted average of 16 numerical factors such as average class size, acceptance rate (fraction of applicants who are admitted), and amount of alumni giving. Trouble is, any combination of these factors just isn't going to change enough from year to year to keep things interesting.

So how on earth can U.S. News explain Caltech's one-year rise?

The magazine tries to deny that there's anything odd about a college improving so quickly. The "best colleges" story argues: "Caltech has always been within striking distance of the top of the chart. In 1989, Caltech was the No. 3 school, ahead of Harvard. ... Last year, Caltech had the fourth-highest score among national universities." The first assertion is irrelevant: We're not interested in the 10-year rise from third but rather in the one-year rise from ninth. The second assertion is technically true but practically dishonest: Caltech had the "fourth-highest score" last year only because there were two three-way ties and one two-way tie among the eight schools that beat it.

But the real reason Caltech jumped eight spaces this year is that the editors at U.S. News fiddled with the rules. The lead story of the "best colleges" package says that a change in "methodology ... helped" make Caltech No. 1. Buried in a sidebar is the flat-out concession that "[t]he effect of the change ... was to move [Caltech] into first place." No "helped" about it. In other words, Caltech didn't improve this year, and Harvard, Yale, and Princeton didn't get any worse. If the rules hadn't changed, HYP would still be ahead. If the rules had changed last year, Caltech would have been on top a year earlier.

(In fact, if the U.S. News criteria are taken seriously, and if they held steady, Caltech may actually have slipped in quality this past year. Most indicators did not change compared to last year. But graduation rate, number of classes with fewer than 20 students, and percentage of faculty members who work full-time actually declined. Only two indicators showed small improvements: percentage of accepted students in top 10 percent of their high-school classes went from 99 percent to 100 percent [big deal!], and Caltech's acceptance rate fell from 23 percent to 18 percent.)

U.S. News denies that it changes the rules--as it does every year--simply to change the results. Robert Morse, U.S. News' statistical guru, explained to me that this year's ranking procedures are an "improvement" over last year's. Doesn't that imply, I said, that last year's rankings were inferior? And shouldn't U.S. News apologize to anyone who made "one of the most important decisions of your life"--possibly turning down Caltech for Princeton--based on rankings the magazine itself now regards as inaccurate? Morse replied that he hadn't said the earlier ratings were inferior. But if something improves, I pressed him, doesn't that mean that it was less excellent before the improvement? Morse grudgingly allowed that I was free to make that inference.

I can't prove that U.S. News keeps changing the rules simply in order to change the results. But if not, U.S. News ought to shy away from horse-race headlines such as "Caltech Comes out on Top." A more honest summary might be "We Finally Realize That Caltech Is Tops." Or "Caltech on Top (Until We Fiddle With Rules Again)."



Talking White

Black people’s disdain for “proper English” and academic achievement is a myth.

Hong Kong’s Protesters Are Ridiculously Polite. That’s What Scares Beijing So Much.

The One Fact About Ebola That Should Calm You: It Spreads Slowly

Operation Backbone

How White Boy Rick, a legendary Detroit cocaine dealer, helped the FBI uncover brazen police corruption.

A Jaw-Dropping Political Ad Aimed at Young Women, Apparently

The XX Factor
Oct. 1 2014 4:05 PM Today in GOP Outreach to Women: You Broads Like Wedding Dresses, Right?

How Even an Old Hipster Can Age Gracefully

On their new albums, Leonard Cohen, Robert Plant, and Loudon Wainwright III show three ways.

How Tattoo Parlors Became the Barber Shops of Hipster Neighborhoods

This Gargantuan Wind Farm in Wyoming Would Be the Hoover Dam of the 21st Century

Oct. 1 2014 8:34 AM This Gargantuan Wind Farm in Wyoming Would Be the Hoover Dam of the 21st Century To undertake a massively ambitious energy project, you don’t need the government anymore.
  News & Politics
Oct. 1 2014 7:26 PM Talking White Black people’s disdain for “proper English” and academic achievement is a myth.
Buy a Small Business
Oct. 1 2014 11:48 PM Inking the Deal Why tattoo parlors are a great small-business bet.
Oct. 1 2014 6:02 PM Facebook Relaxes Its “Real Name” Policy; Drag Queens Celebrate
  Double X
The XX Factor
Oct. 1 2014 5:11 PM Celebrity Feminist Identification Has Reached Peak Meaninglessness
  Slate Plus
Behind the Scenes
Oct. 1 2014 3:24 PM Revelry (and Business) at Mohonk Photos and highlights from Slate’s annual retreat.
Brow Beat
Oct. 1 2014 9:39 PM Tom Cruise Dies Over and Over Again in This Edge of Tomorrow Supercut
Future Tense
Oct. 1 2014 6:59 PM EU’s Next Digital Commissioner Thinks Keeping Nude Celeb Photos in the Cloud Is “Stupid”
  Health & Science
Oct. 1 2014 4:03 PM Does the Earth Really Have a “Hum”? Yes, but probably not the one you’re thinking.
Sports Nut
Oct. 1 2014 5:19 PM Bunt-a-Palooza! How bad was the Kansas City Royals’ bunt-all-the-time strategy in the American League wild-card game?