Heritage Report is latest in series of misleading analyses

Contact:
Madhabi Chatterji, (212) 678-3357; mb1434@columbia.edu
William Mathis, NEPC, (802) 383-0058; William.Mathis@colorado.edu

BOULDER, CO (November 30, 2010) – Did a collection of Florida education policies—ranging from grade retention to school choice and virtual schools—improve achievement and narrow the test-score gap? A recent Heritage Foundation report is part of a larger campaign to convince us that the answer to that question is “yes”—but a new review finds fundamental flaws in the Heritage report that render its conclusions untenable.

The Heritage report, Closing the Racial Achievement Gap: Learning from Florida’s Reforms, was reviewed for the Think Twice think tank review project by Madhabi Chatterji, a professor of educational measurement and evaluation at Teachers College at Columbia University who has conducted considerable research of Florida schools. The review is published by the National Education Policy Center, housed at the University of Colorado at Boulder School of Education.

The Heritage report, authored by Matthew Ladner and Lindsey Burke, contends that Florida’s “far-reaching” education policies have caused test scores to increase and the achievement gap to narrow. In particular, the report focuses on fourth-grade reading scores on the National Assessment of Educational Progress (NAEP). These research claims are also made by Dr. Ladner, the Vice President of Research at Arizona’s free-market Goldwater Institute, in a dozen other reports and articles similar to the Heritage report reviewed by professor Chatterji.

The claims, however, do not withstand scrutiny. “The report’s key conclusions are unwarranted and insufficiently supported by research,” Chatterji states in her review. Most importantly, she points out the very direct effects of the state’s grade-retention policy, causing the report’s comparisons to be largely meaningless. By analogy, consider growth in height instead of growth in test scores. If two states wanted to measure the average height of their fourth graders, but one state (Florida) first identified the shortest 20% of third graders and held them back to grow an additional year before measurement, the study’s results would not be useful.

That, in brief, is the key problem that professor Chatterji identifies with the Heritage report. Florida’s retention policy, instituted in 2002, focuses on third graders, who are held back when their reading scores are low. The Heritage report focuses on NAEP fourth grade reading scores. Low scoring readers—mostly black and Hispanic—were screened out of grade 4 tests, which resulted in inflated and erroneous fourth-grade scores. “Chatterji’s review explains very clearly why the simplistic comparison of fourth graders before and after Florida’s grade retention policy is a predictable and worthless exercise,” says Kevin Welner, professor of education at the University of Colorado and the director of NEPC.

The review also points out that NAEP scores at other grade levels and even NAEP scores in fourth-grade math do not show the same jump as NAEP fourth grade reading. That is, the report cherry-picks the best data. Also, even if scores in Florida are in fact increasing, the report’s methods are too weak to allow for a causal inference. The report uses only descriptive test score trends to compare states and then make sweeping generalizations. Moreover, many other changes occurred in Florida during the period analyzed, including the phasing in of one of the nation’s most ambitious class-size reduction reforms—yet the report never mentions these other possible causes of any improvements.

“In sum, the report’s analyses are highly biased and of very limited value,” Chatterji concludes. “The major elements of Florida’s education reform policies are in need of continuing and more careful examination, individually and collectively, before they can be recommended for wider policy adoption.”

“Notwithstanding the clear problems with Dr. Ladner’s analysis, he has done an excellent job in marketing,” says Welner. “He has repackaged his analysis for at least seven individual states, showing each state’s purported failings relative to Florida. This is how policy gets made—and it isn’t pretty.”

Find Madhabi Chatterji’s review on the NEPC website at:
http://nepc.colorado.edu/thinktank/learning-from-florida

Find Closing the Racial Achievement Gap: Learning from Florida’s Reforms by Matthew Ladner and Lindsey Burke on the web at:
http://www.heritage.org/Research/Reports/2010/09/Closing-the-Racial-Achievement-Gap-Learning-from-Floridas-Reforms

The Think Twice think tank review project (http://thinktankreview.org), a project of the National Education Policy Center, provides the public, policy makers, and the press with timely, academically sound, reviews of selected think tank publications. The project is made possible in part by the generous support of the Great Lakes Center for Education Research and Practice.

The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence.  For more information on NEPC, please visit http://nepc.colorado.edu/.

This review is also found on the GLC website at http://www.greatlakescenter.org/