Noting the nation’s renewed attention to remedying school segregation, Segregation, Race, and Charter Schools presents evidence about the extent of school segregation and its relationship with improving student achievement for students of color. The report argues that school segregation has remained flat for decades and also argues that the nation would be wise to instead attend to improving the quality of schools that students of color and low-income students attend. It points to some urban charter schools as exemplars of this latter approach. However, this review finds that the report omits significant research directly related to the topic and includes other studies that are less relevant. Moreover, the report draws questionable conclusions from studies that are included—conclusions that are not reflective of the research consensus. The report’s selective interpretation of existing research leads to two erroneous conclusions about improving educational outcomes for students of color: (1) that focusing on school integration is relatively unimportant; and (2) that attending to school quality via school choice, rather than addressing the complex array of policies to combat racial segregation, should instead be pursued. In fact, because most forms of school choice further segregation, the report’s recommendation will likely only further segregation and inequality for students.

Two new papers from researchers at the University of Arkansas predict the budgetary consequences of terminating the Louisiana Scholarship Program (LSP), a voucher program that funds over 7,100 Louisiana students to attend private schools. Using an economic model, the papers offer several different scenarios and then conclude that terminating the LSP would increase the costs statewide and do so in almost all districts in the state. The papers’ findings are reasonable but do not make a fully convincing case that the state will incur extra expenditures without the LSP. There may be savings or additional expenditures, depending on several key parameters which have not been precisely estimated. Puzzlingly, the reports’ findings of extra costs run counter to those of the state’s Legislative Fiscal Office, but the reports do not mention this contrary evidence. In any case, the net fiscal effect of terminating the LSP is unlikely to be large; in the context of a $9 billion state expenditure, the change would likely be less than $10 million.

A report published by the Progressive Policy Institute calls for aggressively closing more public schools and expanding charter schools and charter networks. It highlights reforms adopted by Denver Public Schools, notably a “portfolio model” of school governance, and argues that these reforms positively impacted student test scores. However, causality cannot be determined, and the report did not attempt to isolate the effect of a multitude of reforms—including charters, performance pay, and a new performance framework—from larger complex forces shaping student demographics in the city. Written in a reportorial voice, the only data presented are in the form of simple charts. The lack of conventional statistical analyses thwarts the reader’s understanding. The report also characterizes the reform’s adoption as a “political success” born of a healthily contentious electoral process. In doing so, it downplays the role of outside forces and moneyed groups that influenced the form of reforms, and it disregards missed opportunities for meaningful engagement with community stakeholders. Finally, while the report acknowledges the district’s failure to close achievement gaps and admits limitations with the evaluation system, it never explains how a successful reform could generate a widening gap in performance between student groups by race and class.

Update: David Osborne, the report’s author, has posted a response to the review. The response can be found at: http://www.progressivepolicy.org/blog/response-national-education-policy-center/

Terrenda White’s rejoinder to the author's response to her review is posted immediately below the review.

Measuring School Turnaround Success contends that the lack of a shared definition for turnaround success makes it hard for reformers to learn from each other and determine true success. The report claims to develop a model for defining turnaround success with measures, metrics, and cut scores that reach beyond student achievement tests, to include indicators of engagement by students, parents and teachers, teacher and leader effectiveness, and short-term learning outcomes. Unfortunately, given the dearth of research evidence and sound methodological techniques incorporated into its analysis, as well as the omission of several rigorous, peer-reviewed studies that contradict the majority of its proposals, the report does not meet a minimal standard of evidence to support its claims. Ironically, the report focuses largely on standardized test scores, despite its stated intentions. The result is a report that distracts attention and potential resources from schools’ other goals, including civic, social, emotional, and broader academic ones. In the end, the report puts forth yet another proposal, funded by public dollars, that encourages state leaders to continue over-relying on flawed, test-centered strategies. Policymakers and practitioners looking for guidance on measuring turnaround success will not find worthwhile recommendations. Instead, they will encounter several unsubstantiated ones and others that are contradicted by solid peer-reviewed research.

The authors of this report created “efficiency scores” for Milwaukee schools and, on the basis of those scores, draw conclusions about the relative efficiency of Milwaukee’s charter and traditional public schools. A first set of scores was generated by dividing test scores in math and science by per-pupil funding. In a second analysis, “efficiency scores” were estimated for school types using a regression model that also accounted for selected student demographic characteristics. Several major problems arise from the authors’ approach: First, test scores do not comprehensively represent the purposes of schools. Second, threats to the validity of the author’s assumption that there is uniform financial accounting across schools and types are not addressed. Third, the analytic description of the study was incomplete, making interpretation difficult. Fourth, the report makes strong but unmeasured claims about the superior “efficiency” of charter schools based on their having greater autonomy. Finally, the authors did not appear to adjust for selection effects – effects that would prove fatal to their conclusions.  As a result of its manifold flaws, and because its conclusions are not supported by the evidence presented, the report is of little if any use to policymakers.

Update: Will Flanders and CJ Szafir, the report's authors, have posted a response to the review. The response can be found immediately below Cobb's original review of their report and at: http://www.will-law.org/will-blog-will-responds-criticism-colorado-group/.

Casey Cobb's rejoinder to the authors' response to his review is posted immediately below the authors' response.

Two reports claim to offer empirical support for the efficacy of voucher programs that allow parents to use taxpayer dollars to send their children to private schools. One report (A Win-Win Solution) is the latest in a series from the Friedman Foundation for Educational Choice. The Friedman report reviews studies purporting to show positive impacts from voucher programs in the US. The other report (The Participant Effects of Private School Vouchers across the Globe) is from the Department of Education Reform at the University of Arkansas. The authors of the Arkansas report conducted a limited meta-analysis of US and international studies of voucher programs. The two reports share a positive view of the impacts of vouchers, and both focus on randomized studies of those effects. Both reports are marred by a number of serious problems and errors, including misrepresentations of the research literature, a failure to acknowledge the limitations of their approaches, not addressing the shortcomings of the theoretical underpinnings of vouchers, and the use of methods that bias the selections of the studies they utilize. The Friedman report is a rudimentary “vote-counting “analysis of an extremely narrow set of 18 studies using a biased counting system. The Arkansas meta-analysis aspires to be “global,” but despite identifying over 9,000 potential studies for the analysis, ultimately uses only 19, almost half of which were conducted by the Arkansas authors or their associates. Moreover, the “global” meta-analysis only encompasses three countries (one of which is consistently misspelled). Together, their manifold serious flaws undercut the trustworthiness and usefulness of these reports.

A report from Michigan’s Mackinac Center asserts that there is little or no relationship between student achievement and marginal increases to what the report characterizes as the already “high” levels of spending in that state. Yet the report never substantiates its assertion that present spending levels are high, on average, or uniformly high across all children, districts, or schools statewide. The report discounts a significant body of peer-reviewed research that specifically shows positive effects of previous Michigan school finance reforms, including positive effects on state assessments and educational attainment, concentrated on those students who attended, before those reforms, the lowest funded schools or lower performing schools. Additionally, while the report argues that increased spending on schools as they presently exist would necessarily be inefficient and ineffective, this contention is undermined by the lack of evidence for more efficient alternatives and by existing research validating the value of traditional resources. Both a major national study and a Michigan-specific study show funding increases as efficacious when allocated primarily toward traditional investments (increased teacher salaries and smaller class sizes). Finally, the empirical analysis included in the report lacks depth and rigor when compared to four other studies—three of which were peer-reviewed—each of which find positive effects of prior school finance reforms in Michigan.

This report uses state-level data from the National Assessment of Educational Progress to describe a positive association between tracking in eighth grade and the proportion of students passing AP exams in high school. The relationship is moderately strong and holds true for White, Black, and Hispanic students. The report suggests that the separate learning environments for high achievers created by tracking are important for providing students (including students of color) with the skills and knowledge to succeed with the most demanding coursework offered in high schools. The findings are based on correlations and cannot establish a causal relationship, nor can they identify what mechanisms might be at work. However, they are consistent with prior research that has frequently (although not always) identified benefits of tracking for high-achieving students. A key weakness is that the report neglects to consider how tracking is likely to affect lower-achieving students. Tracking is often implemented in ways that hinder the learning of students assigned to low tracks. Because disadvantaged and minority students are disproportionally assigned to low tracks, the report’s conclusion that tracking could be “a potential tool for promoting equity” is dubious.

Relative to earlier research, this study from the Center on Reinventing Public Education provides a more in-depth analysis of policy features across the 27 states that allow online charter schools. It presents a well-organized description of policy features and includes a set of policy recommendations that generally, but not always, follow well from the study’s evidence. Because the findings are largely negative, this report may be seized on by groups that are broadly critical of charter schools. But the report is published by an organization that often advocates for charter school growth, and the authors’ discussion of findings suggests that the charter school establishment is advocating for separating online charter schools from brick-and-mortar charter schools and governing them with a separate policy framework. While there is some justification for such an approach, online charter issues are not completely distinct from issues that arise from other charters. Overall, the detailed analyses of policy environments and the summary of problems in the online charter school sector included in this report should be useful to policymakers who are willing and able to pursue more restrictive oversight and increase accountability for online charter schools.

In a report, University of Arkansas researchers studied the relationship between school vouchers and crime. Using data from the Milwaukee Parental Choice Program (MPCP), they compared crimes processed through the Wisconsin courts for program participants and a matched sample of public school students. Based on a series of comparisons, it finds that some groups of MPCP students are less likely to commit crimes as adults. This result is plausible: education and crime are often found to be negatively correlated and the MPCP has generated some, albeit modest and mixed, benefits. However, the study’s title should not imply that voucher programs are a "get-out-of-jail" card, and the evidence in the study is not causal. One concern is that the paper employs a matching method that omits some important factors that explain school choice and crime. Also, the results are highly variable, with most of the association between MPCP participation and measures of adult crime showing statistically insignificant results. Indeed, a valid interpretation of the paper is that vouchers and crime are not correlated. Conversely, for subgroups and estimation approaches that do yield statistically significant associations, the MPCP effects appear to be too extreme. Even assuming that vouchers do reduce adult crime, it remains unclear by what mechanisms vouchers might do so.

As part of an ongoing series of reports by the National Council on Teacher Quality (NCTQ), Learning About Learning: What Every New Teacher Needs to Know makes broad claims about teacher education based on a limited analysis of textbooks and syllabi. The report argues that teacher education materials, specifically educational psychology and methods textbooks, are a waste of funds and do not adequately focus on what the report identifies as six essential strategies. These inadequacies, the report contends, result in ill-prepared teacher candidates lacking in “research-proven instructional strategies” (p. vi). The report offers recommendations for textbook publishers, teacher education programs, and state departments of education. However, it is not grounded in a comprehensive examination of the literature on teaching methods, and it fails to validate the evaluative criteria it employs in selecting programs, textbooks, and syllabi. The single source it relies on to justify its “six essential strategies” provides limited support for NCTQ’s claims. This primary source concludes, with only one exception, that the evidence supporting each of the six strategies is only moderate or weak. Limiting the analysis to one source that provides only tepid support renders the report of little value for improving teacher preparation, selecting textbooks, or guiding educational policy.

The Manhattan Institute's SchoolGrades.org evaluates and assigns grades, using reading and math test scores, to U.S. schools and compares schools across their respective states and to other countries. They apparently use a four-step process: (1) average two state test scores; (2) “norm” these results to the NAEP exam; (3) make an adjustment to this national “normed” measure using free and reduced price lunch data to account for SES; and (4) “norm” these results to the international PISA exam. The claim is that this process allows a parent to compare a local school to schools in their state and to other countries like South Korea and Lithuania. But the unsubstantiated norming chain is too tenuous and the results are overly extrapolated to be of any useful value. The website does not explain how international scores are “normed” (equated) to the national standard they developed or how letter grades were determined, nor does it explain how free and reduced price lunch counts are used to make socioeconomic adjustments. While there is considerable equating research available, none is cited. Further, the reliance on aggregated test scores is far too narrow a base to serve as a useful evaluation of schools. Thus, the website’s approach to evaluating schools fails on technical grounds and, just as importantly, it fails to understand and consider the broader purposes of education in a democratic society.

This Center for American Progress report examines whether states’ adoption of standards-based policies predicts low-income students’ NAEP achievement trends in fourth and eighth grade math and reading throughout the 2003-2013 decade. The report claims to analyze changes across five separate two-year intervals, but it only reports findings for 2009-2011, with no explanation of why or any documentation of the representativeness of that single interval. The reported finding for the selected interval is that state adoption of standards policies positively predicts fourth (but not eighth) grade math NAEP and eighth (but not fourth) grade reading NAEP. Even these selected positive results are statistically significant only at the generally unacceptable 0.10 level of significance. The report includes effect sizes but nothing about the percent of the variance explained in their model. In short, the report does not adequately describe variables or analytic methods or completely report findings, and the data and methods used do not allow for any causal findings. They use state standards adoption across grade levels and subject areas as well as selected accountability mandates as predictors but fail to assess their quality or fidelity of implementation. Yet based on these very problematic and limited analyses, the authors conclude that their analysis “strongly supports the potential of the Common Core to drive improvements in educational outcomes.” The study simply does not support this conclusion or the set of recommendations that follow. 

A report investigates the enrollment and achievement of students with disabilities and students with English language learning (ELL) needs in oversubscribed charter schools in Boston. The report concludes that Boston charters and Boston Public Schools enroll similar numbers of both special populations and that charter attendance has a positive statistically significant effect for those who enter Boston’s charter school lottery and then enroll after being offered a seat. This review finds that econometric models used to estimate the effects are appropriate, but also more limited than the report would suggest. The report finds some interesting patterns that deserve further study; however the effects cannot be generalized to charter schools outside Boston or even to most students inside Boston. The study also offers no context to compare the size of reported gains and it does not adequately examine how or why the reported test score gains are realized; for example, it does not account for peer effects or spending differences. Ultimately, while this report takes an important step in studying how oversubscribed charters may affect the academic achievement of special needs students, a closer examination is needed in order to accurately inform those making education policy. 

An evaluation report from RAND focused on school-wide initiatives funded by the Bill & Melinda Gates Foundation to promote teaching approaches touted as personalized learning. These reforms generally rely on digital technology and encompass a range of strategies, such as developing learner profiles with individualized goals, and using data to provide personalized learning paths in which students have choice, get individualized support, and engage in learning outside school. The research, which includes many high-quality elements, suggests that some of the studied approaches are associated with higher scores on a common assessment (the MAP). Broad conclusions about the efficacy of technology-based personalized learning, however, are not warranted by the research. Limitations include a sample of treatment schools that is unrepresentative of the general population of schools, the lack of a threshold in the study for what qualified as implementing “personalized learning” in the treatment schools, and the reality that disruptive strategies such as competency-based progression, which require the largest departures from current practice, were rarely implemented in the studied schools.

This report from the Center for American Progress offers 10 recommendations for improving the public perceptions of and experiences of classroom teachers. While elements of these recommendations would likely be beneficial, they also include policy changes that would increase surveillance of teachers, reduce teachers’ job security, evaluate teachers by students’ test scores, and create merit pay systems that would likely have the opposite effect. For evidence, the report relies too heavily on popular rhetoric, sound bites, opinion articles, and advocacy publications to advance a policy agenda that in many ways could do further harm to the teaching profession. However, many of the report’s recommendations do align with policy reforms currently being proposed for the Higher Education Act and included in the Elementary and Secondary Education Act reauthorizations and are therefore important to read critically and consider carefully. In advancing evaluation of teachers by test scores, the report goes against the cautions and guidelines recently released by the American Statistical Association and the American Educational Research Association. Other than a review of contemporary issues, the report offers little of substance to advance the teaching profession.