What would it take for a well-regarded institution—such as the University of Rochester, and a few dozen more like it—to be among U.S. News & World Report’s top 20 national universities? Hundreds of millions of dollars and a prayer, according a new peer-reviewed paper co-written by a former Rochester provost and his staff.
The study, published by the journal Research in Higher Education, argues that small movements in the rankings are simply “noise” and that any kind of sustained upward movement is both immensely expensive and nearly impossible.
Ralph Kuncl, a former Rochester provost who is now president of University of Redlands, in California, co-wrote the paper, which was a decade in the making. He started thinking about changes in the rankings when he was vice provost at Johns Hopkins University.
He said “the trustees would go bananas” when Johns Hopkins dropped in the rankings. The administration would then have to explain what had happened. “Every year Hopkins went from 15 to 16 to 15 to 16—and I thought, ‘What a silly waste of energy,’ ” Kuncl said in an interview Monday. (Johns Hopkins is currently No. 12.)
The paper found that small movements up or down in the rankings are more or less irrelevant. For most universities in the top 40, any movement of two spots or fewer should be considered noise, the paper said. For colleges outside the top 40, moves up or down of four spots should be thought of as noise, too. “For example, a university ranked at 30 could be 95 percent conﬁdent that its rank will fall between 28 and 32, and only when the rank moves beyond those levels can a statistically signiﬁcant change be claimed,” the paper said.
Colleges that want to move any further have a hard row to hoe, the paper argues, because “meaningful rank changes for top universities are difﬁcult and would occur only after long-range and extraordinarily expensive changes, not through small adjustments.”
Robert Morse, who directs the rankings at U.S. News, said the rankings were designed for consumers, not higher-ed administrators. A change in a college’s fundamentals is borne out over time, he said, “like going up stairs or going down stairs—you move up a few—it's not any given year that shows the meaningful change, it's whether over a period of years that the school is moving either upward or downward.”
But the paper, by authors who have all worked at Rochester, adds to the ruminating by administrators over the U.S. News list with several arguments, including in-depth exploration of what it would take for Rochester, consistently in the mid-30s on the list, to break into the top 20. Emory University, Georgetown University, and the University of California at Berkeley currently tie for No. 20.
The researchers examined a decade of data on colleges that were ranked among the top 200 in 2012 and had been among the top 200 ranked colleges for five or more of the last 10 years. While the formula for the U.S. News rankings is available, only some of the data that goes into the rankings is public, so the study tried to closely approximate what U.S. News uses for its calculations.
If it wanted to move into the top 20, Rochester would have to do a lot on several of the various factors U.S. News uses to rank colleges. To move up one spot because of faculty compensation, Rochester would have to increase the average faculty salary by about $10,000. To move up one spot on resources provided to students, it would have to spend $12,000 more per student. Those two things alone would cost $112 million a year.
To get into the top 20, Rochester would also have to increase its graduation rate by 2 percent, enroll more students who were in the top 10 percent of their high school graduating classes, get more alumni to give, cut the acceptance rate, and increase the SAT and ACT scores of incoming students. Some of those things, like offering aid money to highly qualified students, might further increase the expense.
But that’s not all, the paper argues. Rochester would still have to do well in the rankings magazine’s “beauty contest.”
Because 15 percent of the ranking is based on reputation among other administrators, even massive expenditures year after year and huge leaps in student quality and graduation would not be enough. The reputation score as judged by its peers would need to increase from 3.4 to 4.2 on a scale of 5, something that has only a .01 percent chance of happening, the paper said.
Rochester is among the four colleges in the U.S. News national rankings that have had the same reputational score over eight years. The others are Harvard University, the Massachusetts Institute of Technology, and Stanford University. It’s unclear what drives administrators from across the country to consistently consider Rochester to be about the 35th-best college in America. While those four stay exactly the same, most at the top also don’t move much either.
“If all of these changes were made, but a corresponding change in undergraduate reputation did not follow, the second simulation showed that a rank between 25 and 30 would be more typical, and this university would never move into the top 20,” the paper concluded.
Kuncl said institutions spend tens of thousands of dollars on the “stupidity” of sending glossy brochures to other colleges’ administrators to try to beef up reputational scores.
Morse said colleges can rise without a change in their reputation because reputation among higher-ed administrators and high school counselors makes up less than a quarter of the overall rankings. But, he said, “Is it true that the reputational part of the ranking is more stable over time? Yes.”
A spokesman for the University of Rochester said officials have not yet had time to familiarize themselves with the paper. He noted, however, that the current strategic plan does not refer to rankings.
Kuncl said the paper’s findings are likely to apply to other U.S. News lists, including the lists of regional colleges and liberal arts institutions, but that the study focused only on the magazine’s main national universities list.
Some institutions have made rankings part of their stated goals, including Arizona State University, which tied its president's pay to rankings, or Northeastern University in Massachusetts, which mentions the rankings in its strategic plan. Both are among a handful of institutions that have seen sustained increases in their rankings over the past decade.
“We have long said that the rankings are a byproduct of our success, not the real success,” said Northeastern spokeswoman Renata Nyul.