What Women Really Think
Dec. 27 2010 12:09 PM

This NYT article about ranking teachers based on students' test scores-what's known as value-added ranking-is the first thing I've read that really lays out the tensions about the meaning of this data. It made me a lot more nervous about releasing the information publically right now, as a dozen news organizations including the Times have sued to do, in a case that's pending. The rankings sound like they have the potential to become a really useful tool, but also like they are not ready for prime time yet. If the data are released now and unfairly hurt some teachers' reputations, could the backlash prevent the rankings from developing into what we really need?

Value-added rankings are supposed to tell us how much the quality of an individual teacher contrributed to boosting the scores of his or her students. Researchers say that with confidence, they can tell us that a teacher who is in the bottom 10 percent year after year is doing a bad job and a teacher in the top 10 percent is doing a good job. But about the middle, they tell us much less:

"In math, judging a teacher over three years, the average confidence interval was 34 points, meaning a city teacher who was ranked in the 63rd percentile actually had a score anywhere between the 46th and 80th percentiles, with the 63rd percentile as the most likely score. Even then, the ranking is only 95 percent certain. The result is that half of the city’s ranked teachers were statistically indistinguishable."

Another more serious problem for individual teachers: The rankings have a high error rate, which means you need years of data to get a fairly accurate picture:

"One national study published in July by Mathematica Policy Research, conducted for the Department of Education, found that with one year of data, a teacher was likely to be misclassified 35 percent of the time. With three years of data, the error rate was 25 percent. With 10 years of data, the error rate dropped to 12 percent. The city has four years of data."

One more problem: It turns out that the city was using a narrow and too easy test. Now they're trying to fix that, but "Daniel Koretz, a Harvard professor whose research helped persuade the state to toughen standards, said that as a result it was impossible to know whether rising scores in a classroom were due to inappropriate test preparation or gains in real learning." The city won’t have rankings with the higher standards til the next academic year. And would we really need several year’s worth to know anything really useful?

It would be nice to think that the city could release its data now, with a full explanation of the flaws, and everyone would calmly take it for what its worth and refrain from jumping to conclusions about individual teachers. But is that likely? When the Los Angeles Times released this data about L.A. teachers last August, some teachers said they got burned . Do we know about yet to be able to tell whether that cost was worthwhile, on balance? I'm sure it depends who you ask, but it seems to me that value-added rankings, in their current guise, invovle making individual teachers bear a big risk of being misjudged in return for information that could improve the system. Isn't that a lot to ask?

Politics

# Blacks Don’t Have a Corporal Punishment Problem

Americans do. But when blacks exhibit the same behaviors as others, it becomes part of a greater black pathology.

# Lifetime Didn’t Think the Steubenville Rape Case Was Dramatic Enough

So they added a little self-immolation.

Medical Examiner

# The Most Terrifying Thing About Ebola

The disease threatens humanity by preying on humanity.

# The Good Wife Is Cynical, Thrilling, and Grown-Up. It’s Also TV’s Best Drama.

News & Politics
Weigel
Sept. 19 2014 9:15 PM Chris Christie, Better Than Ever
Moneybox
Sept. 19 2014 6:35 PM Pabst Blue Ribbon is Being Sold to the Russians, Was So Over Anyway
Life
Inside Higher Ed
Sept. 19 2014 1:34 PM Empty Seats, Fewer Donors? College football isn’t attracting the audience it used to.
Double X
The XX Factor
Sept. 19 2014 4:58 PM Steubenville Gets the Lifetime Treatment (And a Cheerleader Erupts Into Flames)
Slate Plus
Slate Picks
Sept. 19 2014 12:00 PM What Happened at Slate This Week? The Slatest editor tells us to read well-informed skepticism, media criticism, and more.
Arts
Brow Beat
Sept. 19 2014 4:48 PM You Should Be Listening to Sbtrkt
Technology
Future Tense
Sept. 19 2014 6:31 PM The One Big Problem With the Enormous New iPhone
Health & Science
Medical Examiner
Sept. 19 2014 5:09 PM Did America Get Fat by Drinking Diet Soda?   A high-profile study points the finger at artificial sweeteners.
Sports
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.