Education Policy Research Unit (EPRU) at ASU
Education and the Public Interest Center (EPIC) at CU-Boulder
****NEWS RELEASE--FOR IMMEDIATE RELEASE****
OTHERWISE SOUND REPORT OVERSTATES CLAIMS FOR FLORIDA'S A+ ACCOUNTABILITY SYSTEM
Urban Institute report presents strong analysis of Florida system, but wrongly implies vouchers can be singled out as a cause of school improvement.
Contact: Damian Betebenner, (603) 516-7913; (email) dbetebenner@nciea.org
Kevin Welner, (303) 492-8370; (email) kevin.welner@gmail.com
TEMPE, Ariz and BOULDER, Colo. (Jan. 15, 2008) -- A new study from the Urban Institute's CALDER center attempts to understand whether Florida's accountability system of sanctions and incentives led schools to improve student achievement. The report concludes that the system, including the threat of vouchers, appears to have spurred schools to improve practices and thereby improve performance. A new review of the study praises the data, design and analyses, but the review also identifies several key instances where the report overstates its case.
The CALDER report, "Feeling the Florida Heat? How Low-Performing Schools Respond to Voucher and Accountability Pressure," was authored by Cecilia E. Rouse, Jane Hannaway, Dan Goldhaber, and David Figlio. It was reviewed for the Think Tank Review Project by Damian Betebenner of the National Center for the Improvement of Educational Assessment.
"Feeling the Florida Heat?" challenges the view that schools, threatened with sanctions for not improving student achievement, will respond with only superficial changes that raise test scores but don't contribute to greater learning. The authors conclude that, in Florida, schools receiving a grade of "F" in the state's A+ Plan for Education actually did alter their teaching practices, leading to higher achievement.
The Urban Institute report bases its conclusions on an examination of 35 elementary schools that received an F, making them eligible for sanctions as well as supports and incentives. The "F" schools were given supplementary assistance from the state as well as outside evaluations. Additionally, students in such schools qualified for vouchers, called "Opportunity Scholarships" (an aspect of the reform since declared unconstitutional by the Florida Supreme Court), which allowed those students to transfer to other public and private schools. The researchers who produced the report conducted statistical analyses on the impact of an F, both on student scores and on school policies. The analyses rely on administrative and test-score data as well as surveys completed by principals of approximately 70% of Florida public schools in 2001-2003 and 2003-2004.
"The data sets used for the analyses are unique and impressive given their scope and breadth," Betebenner finds, and they are "adequate to address the research questions posed in the report."
The report notes that schools that are given an "F" rating for two years in a row face stiffer accountability measures, and the authors find that the school scores rise. Such rises have been found in previous research to be driven in large part by superficial measures, such as preparing students to respond to the specific types of questions asked on a given test. The researchers here took two steps to find out if this was the case. First, they analyzed results from a low-stakes exam in addition to the Florida high stakes exam, the FCAT. Second, they analyzed the survey responses of school principals, describing how their school reacted to the F-rating.
Betebenner commends the report for taking these precautions. He observes, however, that the test score increases could still be driven by "unintended policies such as extensive test prep," since the low-stakes exam used (the "Stanford-10") is "highly standardized" and "any skills acquired through teaching to the FCAT such as improved test-taking strategies would likely transfer well to the Stanford-10." Moreover, the data and methods used in this study are not capable of parceling out causal effects to, for instance, changed school practices, test preparation, or any other factors not expressly considered.
"The most prominent shortcoming of the report is its tendency to overstate the predictive relationships indicated by their statistical analyses," Betebenner writes. At times, the report uses appropriate language, describing the relationship between the accountability system and the higher test scores as merely a correlation or association. But Betebenner identifies several instances where the report wrongly slips into causal language: "Yet scattered throughout the paper, the authors write as though the accountability pressure causes the improvement in student achievement." That same incorrect interpretation, he notes, is reinforced by the paper's title.
"[T]here is no supporting evidence that this [causal connection] is true," he writes. "Moreover, even if it is true that the Florida policy of vouchers plus other accountability provisions did lead to the changes in policy and practice, nothing in this new research allows a policy maker to single out either vouchers or other accountability provisions (or a combination) as having such an effect."
Betebenner concludes that the Urban Institute report does offer reason to believe that one or more elements of Florida's accountability systems "may offer ... a lever" to improve student achievement. But he continues, "because changes in school policy and practice can occur for many reasons, this research should not be read to show that the accountability system 'led to' or 'caused' the student achievement increases. Nor does the new report consider whether the accountability levers in Florida are the most effective means of quickly and beneficially transforming the policies and practices of schools in a way that leads to increased student achievement."
Find Damian Betebenner's review on the web at:
http://epsl.asu.edu/epru/ttreviews/EPSL-0801-249-EPRU.pdf
About the Think Tank Review Project
The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.
Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."
CONTACT:
Damian Betebenner
National Center for the Improvement of Educational Assessment, Inc.
Dover, NH
(603) 516-7913
dbetebenner@nciea.org
Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@gmail.com
**********
###
The Education Policy Research Unit (EPRU) conducts original research, provides independent analyses of research and policy documents, and facilitates educational innovation. EPRU facilitates the work of leading academic experts in a variety of disciplines to help inform the public debate about education policy issues.
Visit the EPRU website at http://educationanalysis.org
###
The Education and the Public Interest Center (EPIC) at the University of Colorado, Boulder seeks to contribute information, analysis, and insight to further democratic deliberation regarding educational policy formation and implementation.
Visit the EPIC website at http://education.colorado.edu/epic
###
**********