Reporters covering the presidential horse race this week suffered the rude intrusion of a substantive question. A research paper released by Rand–and aggressively marketed by the Gore campaign–suggests that Texas’ much-ballyhooed gains in educational achievement are illusionary. According to the authors of the report, increases in scores on the Texas Assessment of Academic Skills are contradicted by data from National Assessment of Educational Progress, the widely respected, federally sponsored test. The Bush team responded by pointing to an earlier Rand study that says just the opposite: that NAEP scores show Texas’ educational performance rising faster than other states. What gives?
There are two technical explanations for the disparity. One is that the time frames of the two studies are not congruent. The first study, by David Grissmer, et al., looks at data from NAEP tests spanning the years 1990-96. The second, new study, by Stephen Klein, et al., uses NAEP tests from 1992 to ‘98. The other explanation is that the first study, which was not focused specifically on Texas, adjusts the NAEP data for family characteristics based on census data. The second study, which does focus exclusively on Texas, uses only the test scores themselves and converts them mathematically in order to compare them to TAAS results. These differences in data and methodology yield substantially different verdicts. Both studies concur that Texas has seen educational gains over the past decade. But where Grissmer shows those increases comparing favorably to other states, Klein sees them mostly in line with other states. And where Grissmer cites gains in minority test scores, Klein suggests that the “achievement gap” has actually increased in Texas under Bush.
Still with me? Despite the difference in their conclusions, the two studies don’t have the partisan implications one might expect. There is much to encourage Gore supporters in the Grissmer study, which contends that about two-thirds of the difference between academic performance by similar students in Texas and in California is attributable to smaller class size and greater resources in Texas. In other words, it argues that what’s needed is more education spending of the sort proposed by Al Gore. Moreover, Bush was governor for less than half the time covered by the first study. If its conclusions hold up, the improvement is as much a tribute to his predecessor, Ann Richards, as it is to him.
There is also some comfort for Bush in the second study, which acknowledges that Texas scores have risen disproportionately in fourth-grade math, one of the three categories examined. And if Texas isn’t doing as well as previously thought, that’s hardly good news for Al Gore. As the education scholar Diane Ravitch points out, Texas has implemented precisely the policies encouraged by the Department of Education under the Clinton administration’s Goals 2000 program. In a way, Texas has been pursuing both Gore’s education policy and Bush’s.
But who’s right about the central question of whether Texas’s NAEP scores are rising faster than average? On this issue, the unadorned, unadjusted data support Grissmer and thus Bush. According to Larry Feinberg, the spokesman for the National Assessment Governing Board, Texas has risen faster than average in performance on fourth- and eighth-grade math tests over the full span of years that students have been tested at those levels. In Texas, fourth-grade scores went from 218 to 229 on the NAEP point scale while they went from 218 to only 222 nationally. Eighth-grade math went up 13 points in Texas as compared to eight points nationally over a slightly longer time period. Texas has also risen faster that the national average in reading scores though it’s disputable whether the increase is large enough to be statistically significant. And while NAEP data show that the gap between white and minority performance hasn’t narrowed appreciably, the disparity hasn’t increased as the Klein study suggests.
Texas also performs impressively in categories where NAEP data doesn’t afford the possibility of comparisons over time. Neither the Grissmer nor the Klein study looks at the test of eighth-grade writing that was carried out in 1998. In this category, Texas ranks with Massachusetts, Connecticut, Maine, and Wisconsin in the top tier of states. Nationally, 29 percent of black eighth-graders scored “below basic” in writing. In Texas, only 17 percent did. Nationally, only 7 percent of black students were rated “proficient or above” in writing. In Texas, 20 percent were. It’s impossible to demonstrate that Texas’ improvement in this category is outsized relative to other states because the data doesn’t go back far enough. But for a state with its demographic characteristics, such a high absolute position surely means something. However you slice the NAEP data, Texas looks pretty good.
But there is something else in the new study that should be troubling to the Bush camp. The import of Klein’s comparison is that there’s something screwy about the TAAS, which shows gains four to six times as large as those on NAEP as well as big gains for minorities relative to whites that aren’t substantiated elsewhere. The study attributes this conflict to a system of what it calls “high-stakes testing.” What the authors mean by this term is that schools and teachers are rewarded or punished based on test results. For instance, Texas teachers may get raises based on how their students perform on TAAS. The Klein paper’s plausible surmise is that if much depends on a state-level test, strong “results” may not indicate real progress. The incentives can lead to bad practices, like teaching to the test and even corrupt ones, like cheating. In short, if you want to an objective, reliable assessment of how schools are doing, you should use NAEP.
The reason this matters in the campaign is that Bush’s education plan favors state tests like TAAS over the national one. His proposal would award federal bonuses to states based on their performance on their own less reliable tests. These results are supposed to be “verified” by NAEP, but it’s not clear how such verification would work. At the very least, it’s a gratuitously complicated, multilayered system of accountability. The new Rand study doesn’t undermine Bush’s education record in Texas. But it does make an implicit case that there’s a flaw in his education plan.
