Here’s a short section from one of my papers currently in progress (part of the summary of existing literature on alternative models/strategies, and marginal expenditures).

A series of studies from Roland Fryer and colleagues have explored the effectiveness of specific charter school models and strategies, including Harlem Childrens’ Zone (Dobbie & Fryer, 2009), “no excuses” charter schools in New York City (Dobbie & Fryer, 2011), schools within the Houston public school district (Apollo 20) mimicking no excuses charter strategies (Fryer, 2011, Fryer, 2012), and an intensive urban residential schooling model in Baltimore, MD (Curto & Fryer, 2011).  In each case, the models in question involve resource intensive strategies, including substantially lengthening school days and years, providing small group (2 or 3 on 1) intensive tutoring, providing extensive community based wrap around services (Harlem Childrens’ Zone) or providing student housing and residential support services (Baltimore).

The broad conclusion across these studies is that charter schools or traditional public schools can produce dramatic improvements to student outcomes by implementing no excuses strategies and perhaps wrap around services, and that these strategies come at relatively modest marginal cost. Regarding the benefits of the most expensive alternative explored – residential schooling in Baltimore (at a reported $39,000 per pupil) – the authors conclude that no excuses strategies of extended day and year, and intensive tutoring are likely more cost effective.

But, each of these studies suffers from poorly documented and often ill-conceived comparisons of costs and/or marginal expenditures.

In their study on the effectiveness of no excuses New York City charter schools, Dobbie and Fryer (2011) use data on 35 [those responding to their survey] charter schools to generate an aggregate index based on five policies including teacher feedback, use of data to guide instruction, high-dosage tutoring, increased instructional time and high expectations. [i] They then correlate this index with their measures of school effectiveness across the 35 schools, finding a significant relationship. Separately, the authors report weak or no correlations between “traditional” measures of school resources including per pupil spending and class size and their effectiveness measures, concluding that these measures are not correlated with effectiveness. In short, Dobbie and Fryer argue that potentially costly strategies matter, but money doesn’t. [or so the headlines went]

First, if potentially costly strategies matter (even if those costs are never measured), then so too does money itself. Second, the authors’ analysis and documentation of the financial data is woefully inadequate.[ii] The authors fail entirely to consider that the majority (55 to 60%) of per pupil spending differences across New York City charter schools are explained by grade ranges served and total enrollments (and/or enrollment per grade level, economies of scale), where enrollment is to some extent a function of institutional maturation (scaling up) (Baker and Ferris, 2011, p. 33).[iii]  Given the extent that expenditure variation is largely a function of uncontrollable structural differences across these schools, it is unlikely that one will find a simple correlation between spending variation and student outcomes (without finding some way to control for the structural differences). The authors also fail to report the source or descriptive statistics on their expenditure measure.

In earlier work on Harlem Childrens’ Zone, Dobbie and Fryer[iv] similarly argued that the substantial benefits they found for children participating in HCZ charter schools could be obtained at what they [feebly attempt to] characterize as negligible marginal expense. They arrive at this conclusion via the following [hap-hazard] cost calculation and [bogus] comparison:

The total per-pupil costs of the HCZ public charter schools can be calculated with relative ease. The New York Department of Education provided every charter school, including the Promise Academy, $12,443 per pupil in 2008-2009. HCZ estimates that they added an additional $4,657 per-pupil for in school costs and approximately $2,172 per pupil for after-school and “wrap-around” programs.This implies that HCZ spends $19,272 per pupil. To put this in perspective, the median school district in New York State spent $16,171 per pupil in 2006, and the district at the 95th percentile cutpoint spent $33,521 per pupil (Zhou and Johnson, 2008).[v]

Accepting the additional costs of Harlem Childrens’ Zone as adding up to $19,000 per pupil and accepting as a relevant comparison basis that this figure lies somewhere between the New York statewide median and statewide 95%ile of district spending, then the marginal expense for Harlem Childrens’ Zone might just be trivial. But the marginal expense calculation for HCZ is not clearly documented and highly suspect and the comparison basis misleading.

Baker and Ferris (2011) discuss the difficulties of deriving comparable spending per pupil figures for Harlem Childrens’ Zone schools, pointing out that reported total revenues based on IRS filings vary from $6,000 to $60,000 per pupil (p. 13) depending on the year of data and which children are counted in the denominator (charter students or all school aged residents in the zone).

Further it makes little sense to contextualize the HCZ total figure by placing it between the statewide median and 95%ile district, where affluent suburban Westchester County and Long Island districts far outpace per pupil spending in New York City (Baker and Welner, 2010, p.  10).[vi] Rather, more meaningful comparisons might use relevant budget components for all schools in New York City, or schools serving similar student populations in the same area of the city. Using the city Independent Budget Office (2010b) figure for 2008-09 of $15,672, and accepting the authors total cost figure of $19,000 per pupil, the marginal expense for HCZ would be 21%. Comparing against nearby school site budgets for select schools (see Baker and Ferris, p. 24), the marginal expense is 36 to 60%.

Similar imprecision plagues Fryer’s analysis of transfer of “no excuses” strategies from the charter school context to traditional public schools in Houston, Texas. Fryer explains in his study of Apollo 20 schools in Texas:

The marginal costs are $1,837 per student, which is similar to the marginal costs of other high-performing charter schools. While this may seem to be an important barrier, a back of the envelope cost-benefit exercise reveals that the rate of return on this investment is roughly 20 percent 30 – if one takes the point estimates at face value. Moreover, there are likely lower cost ways to conduct our experiment. For instance, tutoring cost over $2,500 per student. Future experiments can inform whether three-on-one (reducing costs by a third) or even online tutoring may yield similar effects.

Among other things, it is important to understand that this $1,837 figure is derived in a Houston, TX context (as opposed to an NYC context) where the average middle school operating expenditure per pupil is $7,911, for an average marginal expense of 1837/7911 = 23.2%.  While no documentation is provided for the $1,837 figure in Fryer’s paper, that figure is quite close to the average difference in current operating expenditure for the 5 Apollo 20 middle schools in Houston compared to all schools in Houston. But, when comparing only to other Houston Middle Schools that figure rises to $2,392, or 30%. In our view, a 23% to 30% increase in cost is substantial, but further exploration of the true costs of scaling the various reform strategies presented is warranted. [data available here:http://ritter.tea.state.tx.us/perfreport/aeis/2010/DownloadData.html]

In short, across Fryer’s various studies, we find a range of marginal expenses for preferred models and strategies from 21% to 60% above average expenditures of other schools not using the preferred models and strategies. So, what are these studies really saying?

Setting aside the exceptionally poor documentation behind any of the marginal expenditure or cost estimates provided in each and every one of these studies, throughout his various attempts to downplay the importance of financial resources for improving student outcomes, Roland Fryer and colleagues have made a compelling case for spending between 20 and 60% more on public schooling in poor urban contexts including New York City and Houston, TX.

I suspect there are more than a few urban superintendents and principals out there who would appreciate seeing and infusion of resources of this magnitude. And many might even be happy to allocate the bulk of those resources to adopt such strategies as increasing teacher compensation in order to extend school days and years and implement intensive tutoring supports (surprisingly non-reformy strategies).

I should also point out that 20% to 60% more funding, while marginally improving student outcomes in these districts, likely still falls well short of providing children attending poor urban districts equal opportunity to achieve outcomes commonly achieved by their more affluent suburban counterparts, and may fall well short of providing adequate resources for these children to gain access to and succeed in higher education and the labor market beyond. Estimating the true costs of these more lofty outcome objectives is a topic for another day.

NOTE: I would caution however, that we have little basis for asserting that a 20 to 60% increase in per pupil spending would be more efficiently spent on these strategies than on such alternatives as class size reduction and/or expansion of early childhood programs. These comparisons simply haven’t been made, and Fryer’s attempt at such a comparison (NYC “no excuses” study) is woefully inadequate.  Pundits who argue that class size reduction is an especially expensive and inefficient alternative seem willing to ignore outright the substantial additional costs of the strategies promoted in Fryer’s work, arriving at the erroneous conclusion (with Fryer’s full support) that class size reduction is ineffective and costly, and extended school time and intensive tutoring are costless and highly effective.