In a new study, Sean Reardon, a professor of poverty and inequality in education at Stanford University, looked at countrywide math and reading test data from children in grades three through eight and found some fascinating and surprising results that call into question how we think about “good” and “bad” schools.
I spoke by phone with Reardon recently. During the course of our conversation, which has been edited and condensed for clarity, we talked about what test scores actually tell us, the limits of his own study, and why the much-maligned Chicago Public Schools district may be stronger than its reputation.
Isaac Chotiner: What is the main thing about this study that surprised you, or that you think we should take away from it?
Sean F. Reardon: The surprising thing is that there are high rates of test-score growth in lots of districts where I think people wouldn’t have expected to see that. Not everywhere, but certainly a number of low-income districts. There’s a number of them that have a lot higher, faster growth than I expected or that I think a lot of people would have expected.
So, is the upshot of that the test scores actually don’t tell us much? Or that we’re looking at the wrong factors when we try to appraise what a district is doing right or wrong or why it might or might not succeed?
I think one upshot is, if you just look at average test scores in a place, you’re getting a combination of two things. You’re getting on the one hand kind of how much reading and math skills the kids have early on. Like, what did they learn at preschool and early elementary school and what do they get from your home environment.
And the other component is sort of how much are they learning during elementary and middle school. And we ought to probably judge school systems more on how much kids are learning while they’re in school rather than how much did they know when they got to school.
It turns out that some places look great mostly because their kids came in with high test scores, but their growth rates aren’t particularly high during elementary and middle school. And other places look good, or better than you’d expect, because kids are learning a lot in elementary and middle school. So the takeaway is, by judging the school systems based on average test scores, we might be confounding what the school systems are really doing with what kind of skills kids have when they come into school.
How exactly did you conduct this study?
Part of the challenge is that, although we test kids in every public school in the United States a lot, we use different tests in different grades, in different years and in every state. So we don’t have common tests that we can use to compare kids across time or place.
So the first thing we had to do was figure out if we could equate those test scores over time and place. And so, that’s what we did first. We used this other test called NAEP, the National Assessment of Education Progress, which is given to samples of kids in every state. And it’s the same test in every state. And so, we can then use it to link the individual state tests to each other via this other test.
And once we figured out how to get all the test scores in different states and grades onto a common scale, then we both computed measures of how much kids know in third grade, how much they know in eighth grade, how fast the scores are changing from third to eighth grade. And then we compare those across places.
What did you learn specifically about Chicago?
Chicago’s sort of fascinating because it’s a very large. It’s the third largest district in the country, it’s relatively low income, and it’s a place that historically people have thought of as a not very … Bill Bennett back in the ’80s called it the worst school system in the country, or something like that. People historically haven’t thought Chicago has a great public school system. And if you look at scores in third grade in Chicago, they’re pretty low. They’re like a grade and a half below the national average. So kids don’t have great math and reading skills in third grade in Chicago.
But by eighth grade they have almost caught up to the national average. They’re about a half grade below the national average. And so, essentially, the average student in Chicago looks like they’re learning six years worth of math and reading skills in the five-year period between third and eighth grade. So that’s 20 percent faster than the national average. And it’s not just a few kids learning 20 percent faster. It’s 370,000 kids in the district, so it’s a lot of kids learning substantially faster than average. And that’s surprising I think. But good news.
Is there a broader case nationally in the districts you studied for why these schools are doing better than we might think? Are there some strategies that they’re using that people didn’t realize?
That’s the million-dollar question. And we don’t know the answer to that yet and that’s really kind of item one on the agenda to figure out. And not just for us. We made all the data public so that other people, researchers, and educators could dig into them and try to figure out can we learn from these about what are the strategies that lead to high rates of learning.
The usual theories people have are things like school systems that are able to attract and retain skilled teachers, that have good school leadership, that have safe school environments, that have a kind of coherent sort of curriculum and instruction standards. Where they’re able to focus on learning and take care of kids’ other needs outside of the classroom so the kids really can focus on learning while they’re in school.
That’s kind of the broad set of things we think about. But practically speaking—what that means and which ways of doing that are most effective—I think we don’t know as much as we need to know.
I suppose this study also suggests that we need much more comprehensive and nuanced ways of looking at test scores, because we shouldn’t be punishing schools if they’re actually performing.
Right. And I totally agree with that. And part of the reason we wanted to do this was to see if looking at growth would give us a kind of a different and better signal of school effectiveness. But I think it’s really important not to overstate the quality of what we did, in the sense that I think our growth measures are better than just sort of status measures. But they’re far from perfect for two main reasons.
One is: We probably should evaluate schools on more than just their math and reading score growth. If you’re a parent, you want your kid to be a lot more things than good at math and reading. You’d like them to know how to read and do math. But you want them to be creative, and be able to work with others well, and be happy, and make friends well, and have critical-thinking skills. You want them to be lots of things, and the math and reading test scores are part of that but far from the sum total of it.
And so, these are imperfect in their scope. And then the other reason these are imperfect is that the way we measure this is, essentially—and this is the very simplified thing of what we do—we take the average eighth-grade scores in one year and we compare them to the average seventh-grade scores the year before, and the sixth-grade scores the year before, and the fifth grade before and so on. But we’re not necessarily comparing exactly the same kids to each other.
We’re in a district where some kids are moving in and out. We might be kind of mischaracterizing the actual student growth because we’re looking at that. We could develop a system that solves that problem better, too.
Listening to you talk makes me think that education is studied so often, and we talk about it all the time, but there’s just so much we don’t know that hasn’t been properly evaluated.
Well, yes and no. I think we’ve developed policies that focus to some extent on not entirely the wrong targets but an incomplete set of targets. And, as a result, a lot of our knowledge about education is not as comprehensive as we want because a lot of the policy incentives are aligned toward average math and reading scores. And that’s a pretty incomplete picture of what we want out of our schools. And we tried to produce something that’s a marginal improvement over that but far from the best and final answer.
