NEPC Resources on Accountability and Testing
Accountability, Rigor, and Detracking
NEPC Review: Feeling the Florida Heat? How Low-Performing Schools Respond to Voucher and Accountability Pressure (Urban Institute, November 2007)
This study examines the relationship between high-stakes school accountability and its effects upon student test scores and school policies. The authors seek to understand the extent to which accountability sanctions and incentives for the poorest-performing schools in Florida explain subsequent changes in school practices and policies as well as achievement — measured by state assessment data, Stanford-10 assessment data and surveys of public school principals. Based on statistical analysis of the lowest-performing schools, the authors report that accountability incentives and sanctions are related to school practice and policy as well as to student achievement. The report uses comprehensive data sources and applies appropriate methodologies to address the research question. Its analyses demonstrate a mediating relationship for school policies between accountability and achievement gains, a finding consistent with both the literature on the subject and common sense. However, the report overstates and makes causal claims about the relationship between accountability sanctions and improvements in school achievement. In this way, the report’s title and some causal statements in the body of the report are unfortunate in that they overstate the report’s sound findings and suggest that vouchers and other accountability measures are shown to be the cause of achievement gains in some of Florida’s lowest-performing schools.
Suggested Citation:
Betebenner, D. (2008). Review of "Feeling the Florida Heat? How Low-Performing Schools Respond to Voucher and Accountability Pressure." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-feeling-florida-heat-how-low-performing-schools-respond-voucher-and-accountability-
NEPC Review: Are Private High Schools Better Academically Than Public High Schools? (October 2007)
Public versus private school achievement gaps in general and the effects of school choice on academic outcomes in particular remain controversial issues. The author reviews two recent reports of empirical studies on this topic: one from the Milton & Rose D. Friedman Foundation (MFF) and the other from the Center on Education Policy (CEP). MFF presents its empirical analysis in the context of the larger policy question about the effect of school choice, whereas CEP simply attempts to answer a research question, with policy implications, about a possible public-private school achievement gap. Both studies contribute new evidence to the existing literature through secondary analyses of national high school student datasets — the Educational Longitudinal Study (ELS) and the National Education Longitudinal Study (NELS) databases. The two reports in tandem provide contrasting views and results regarding private school effects. MFF argues that private schooling is more successful at improving student test scores; CEP argues that public and private schools have relatively equal success. This review provides an independent cross-examination of the two data sources and shows that the public-private high school gaps in math achievement gain scores were almost null (in the NELS) or too small to be practically significant (in the ELS). Therefore, the seemingly divergent findings and conclusions at the first glance may have been largely due to their different interpretations rather than real differences in the results. Both studies could have given more useful guidelines for policy and practice if they had examined reasons for observed gaps (or the lack thereof) between public and private schools.
Suggested Citation:
Lee, J. (2007). Review of "Two Reports Addressing the Achievement of Students in Private and Public Schools." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-two-reports-addressing-achievement-students-private-and-public-schools
NEPC Review: End It, Don't Mend It: What to Do with No Child Left Behind (September 2007)
This new report from the Cato Institute begins with a solid analysis of No Child Left Behind's difficult-to-discern effects on student achievement, concluding that the law has narrowed the curriculum while failing to boost test scores. The report also includes a useful, though one-sided, review of current debates on Capitol Hill, focusing on proposals that the authors believe offer little more than tinkering with the current law. This prompts the question of why major players have yet to back out of this short-term policy quagmire and ask, what would an effective federal role look like? Despite this provocative thinking, the authors ultimately fall back on the Cato creed: shrink the central state and expand market choice in every sector of human activity. The report suffers from two key weaknesses. First, the authors ignore historical evidence showing that state-led accountability efforts, extending through the late 1990s, were associated with significant gains in achievement and narrower racial gaps. Rather than asking how Washington might learn from the states' apparent success, the authors infer from NCLB's limitations that any federal education policy will fail. Second, the authors' failure to subject market-based approaches to the same critical analysis applied to NCLB leads them to endorse a very narrow range of policy alternatives.
Suggested Citation:
Fuller, B. (2007). Review of "End It, Don't Mend It: What to Do with No Child Left Behind." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-end-it-don%E2%80%99t-mend-it-what-do-with-no-child-left-behind
NEPC Review: Portfolios: A Backward Step in School Accountability (September 2007)
This self-described "research study" and following press release are intended to influence the debate over the direction of the reauthorization of NCLB, offering a defense of the current test-based accountability system against the inclusion of "multiple measures." The report presents a review of the research on portfolios in large-scale school accountability systems, concludes that portfolio assessment is severely flawed, and then characterizes portfolios as a proxy for all non-test-based measures of student performance. The report has several glaring weaknesses, however. The literature review cherry-picks two studies, both conducted 13 years ago and, on the basis of those studies, concludes that portfolios are not reliable and are too expensive for large scale accountability systems. Yet other large-scale studies of portfolios – some of which are discussed in one of the two studies that the report itself relies on – come to different conclusions but are not examined or even mentioned. An even bigger problem with this new report (which is repeated in the press release), however, is the author’s decision to present portfolios as somehow representative of all non-test-based measures of student performance – which they clearly are not. This results in a document that is of little value for research or policy development.
Note that an "Issue Brief" (press release), published by the Lexington Institute after the main report, is available at:
http://lexingtoninstitute.org/1169.shtml
Suggested Citation:
Mathis, W. (2007). Review of "Portfolios: A Backward Step in School Accountability." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-portfolios-a-backward-step-school-accountability
NEPC Review: Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind? (June 2007)
A new report released by the Center on Education Policy, "Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?" has received a great deal of attention in the press and is likely to be cited often in the upcoming debate on the reauthorization of the No Child Left Behind Act (NCLB). Using states as their unit of analysis, this report concludes that since the implementation of NCLB in 2002, on average, test scores have increased, the achievement gap has narrowed, and achievement gains post-NCLB have increased faster than before NCLB. Despite its attempt and intent to carefully analyze the complex issue of test score improvement before and after the implementation of NCLB in 2002, however, there are some important weaknesses in the analysis that may have resulted in a much more optimistic picture of the impact of the legislation than the data warrant. The report acknowledges several important methodological weaknesses, but other such weaknesses are never mentioned. Among these additional problems are issues of scope, measurement, and selection—all of which ultimately call into question the robustness of the findings, rendering the report’s conclusions far from definitive.
Suggested Citation:
Yun, J. (2007). Review of "Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?" Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-answering-question-that-matters-most-has-student-achievement-increased-since-no-chi
NEPC Review: State Takeover, School Restructuring, Private Management, and Student Achievement in Philadelphia (February 2007)
In 2002, the city of Philadelphia began a policy of restructuring its lowest-achieving elementary and middle schools. 86 schools were included. Restructuring can take on a wide variety of forms, but in Philadelphia the most prominent approaches shifted school management to either the district or one of several private providers. In 2007, after four years of this policy, two research reports were issued, one by RAND in collaboration with Research For Action (RAND-RFA) and one by the Program on Education Policy and Governance (PEPG). Both reports examined whether any positive effects on the math and reading achievement of students could be attributed to privately managed schools, district-managed schools, or neither. According to the RAND-RFA report, private management has had no cumulative effect on math or reading achievement, while district management has had a positive effect on math achievement but no effect on reading. According to the PEPG report, private management has had a positive effect on the percentage of students reaching "Basic" levels of performance in math and reading, while district management has generally had no effect. The different findings from the two reports can largely be explained by the fact that PEPG did not have the same access to data as did RAND-RFA. PEPG also analyzed data using a different methodological approach than did RAND-RFA, due in large part to the data limitations. This review identifies and describes methodological weaknesses in the report from RAND-RFA as well as in the PEPG report. Overall, while the RAND-RFA study appears to better capture the overall effects of Philadelphia's reform than does the PEPG study, it does not differentiate effects between the elementary and middle school grades. Further analysis and research is needed before drawing any definitive conclusions.
Suggested Citation:
Briggs, D. (2007). Review of Two Reports about Privately Managed Schools in Philadelphia. Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-two-reports-about-privately-managed-schools-philadelphia
The Testing Culture and the Persistence of High Stakes Testing Reforms
The Use of Assessment for Curricular Differentiation
NEPC Review: The State of State Standards 2006 (Thomas B. Fordham Institute, August 2006)
A new Fordham Institute report assigns letter grades to each state for its academic standards. A review of this report finds that the method for determining those grades is flawed.
Suggested Citation:
Howe, K. (2006). Review of "The State of State Standards 2006." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-the-state-state-standards-2006
NEPC Review: On the Public-Private School Achievement Debate (August 2006)
This report claims that private schools outperform public schools. According to a review by the Think Tank Review Project, the report applied inappropriate models to account for the demographic differences between students.
Suggested Citation:
Lubienski, C. and S. (2006). Review of "On the Public-Private School Achievement Debate." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-on-public-private-school-achievement-debate
NEPC Review: The State of High School Education in Wisconsin: A Tale of Two Wisconsins (January 2006)
This report documents "Two Wisconsins" separated by wealth. It concludes the difference in test scores between rich and poor is not attributable to school funding. To close the achievement gap, the author recommends that school districts require students to take more rigorous courses.
Suggested Citation:
Mathis, W. (2006). Review of "The State of High School Education in Wisconsin: A Tale of Two Wisconsins." Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from http://epicpolicy.org/thinktank/review-the-state-high-school-education-wisconsin-a-tale-two-wisconsins