In a previous post, I discussed “Apollo 20,” a Houston pilot program in which a group of low-performing regular public schools are implementing the so-called “no excuses” education model common among high-profile charter schools such as KIPP. In the Houston implementation, “no excuses” consists of five basic policies: a longer day and year, resulting in 21 percent more school time; different human capital policies, including performance bonuses and firing and selectively rehiring all principals and half of teachers (the latter is one of the “turnaround models” being pushed by the Obama Administration); extensive 2-on-1 tutoring; regular assessments and data analysis; and “high expectations” for behavior and achievement, including parental contracts.

A couple of weeks ago, Harvard professor Roland Fryer, the lead project researcher, released the results of the pilot’s first year. I haven’t seen much national coverage of the report, but I’ve seen a few people characterize the results as evidence that “’No excuses’ works in regular public schools.” Now, it’s true that there were effects – strong in math – and that the results appear to be persistent across different model specifications.

But, when it comes to the question of whether “no excuses works,” the reality is a bit more complicated. There are four main things to keep in mind when interpreting the results of this paper, a couple of which bear on the larger debate about “no excuses” charter schools and education reform in general.

First, and most obviously, these results pertain only to the program’s first year. That is not enough time to draw anything beyond extremely tentative conclusions about whether a given intervention “works.” On a related note, it bears mentioning that even the final results will still leave open important common questions such as selection (the evaluation is not experimental) and scalability.

Second, results were basically mixed. The program’s effects were rather strong in math – on average, about 0.25 standard deviations, a substantial effect size, with the magnitude and statistical significance of the estimates varying widely by grade, school and other variables (for example, treatment effects were only marginally significant in middle and most high school grades). But the reading effects were essentially nil. While there was again some variation by grade and other subgroups, the average reading effect was quite small (about 0.05 standard deviations), and only very marginally statistically significant, which means that we can’t have much faith in its precision. So, to whatever small degree this analysis provides tentative evidence as to whether the “no excuses” model “worked,” the evidence is somewhat inconsistent. As is frequently the case, the education policy interventions in fashion these days (especially charter schools), when they work, tend to have meaningful effects on only one of the two commonly-tested subjects. This is in no small part because the factors, such as content knowledge, that determine reading ability are developed before and outside of schools.

The third thing to keep in mind, which I discussed in my prior post about Apollo 20, is the fact that this program – and, by extension, the “no excuses” model in general – is quite expensive. Fryer reports a cost of around $2,000 per pupil (the total annual cost of the program is $19 million), with amounts varying by whether or not students received the tutoring. The overall additional spending represents a 20 percent increase over average per-pupil expenditures. Fryer notes that this dollar amount is similar to the spending of other “no excuses” charters, such as those in New York City (though he provides no citation for his figures on other charters, which are difficult to track [and compare across locations]).

Nevertheless, let’s just say his numbers are correct, and let’s also take a huge leap and say that the achievement return on this spending is worthwhile (as Fryer argues is the case for the math results, using a very rough calculation). This is a massive investment, even if one takes steps to lower the costs (e.g., by using small group instead of 2-on-1 tutoring). The Apollo 20 program requires a roughly 20 percent increase in spending during a time when many states, including California, are actually cutting spending by similar proportions. So, to the extent that the eventual final Apollo 20 results suggest that the “no excuses” model might be expanded into regular public schools (and that’s still an open question), it will require a huge cash infusion.

My fourth point about this paper is a more general conceptual one, which pertains to the “no excuses” philosophy. As I have discussed several times, the best question to ask is not whether “no excuses” (and charter schools in general) works, but why. And this paper only helps a little bit. It gives us five policies to consider, but doesn’t really isolate the unique effects of each of the five policies implemented (for example, by varying them between schools).

There is one partial exception – tutoring. The math gains among Apollo 20 students were in no small part driven by progress in grades (six and nine) that received the 2-on-1 tutoring. Estimated effects in the non-tutored grades were much more moderate, and there was no tutoring in reading, which was largely unaffected. In addition, Fryer estimates directly the effect of the tutoring treatment (math only), which is large and statistically significant. So, this represents decent circumstantial evidence that tutoring is a huge factor in the math results. I have previously speculated that the extended time is also a big one, but I can’t be sure.

In any case, among these five interventions (tutoring, extended time, improving human capital, interim assessments and “high expectations”), only one of them – “improving human capital” through more selective hiring and performance bonuses – focuses directly on improving teacher quality, the primary tool advocated by market-based reformers. Frankly, the human capital component is really the only one that could be called “market-based” by any reasonable definition (though the regular analysis of interim assessment data might be loosely classified as such).

In other words, the teacher-focused, market-based philosophy that dominates our public debate is not very well represented in the “no excuses” model, even though the latter is frequently held up as evidence supporting the former. Now, it’s certainly true that policies are most effective when you have good people implementing them, and that the impact of teachers and administrators permeates every facet of schools’ operation and culture. Nonetheless, most of the components that comprise the “no excuses” model in its actual policy manifestation are less focused on “doing things better” than on doing them more. They’re about more time in school, more instructional staff, more money and more testing. I’ve called this a “blunt force” approach to education, and that’s really what it is. It’s not particularly innovative, and it’s certainly not cheap.

To be clear, I’m obviously not saying that “no excuses” is somehow inconsistent with teacher-focused or market-based reform. Those who consider themselves “reformers” are a diverse group, and they embrace a variety of different approaches.

What I am saying is that, to whatever extent “no excuses” works in regular public schools (and, again, that remains a completely open question), it implies that we might consider focusing a bit more on the conditions and policies of schools, in addition to the people that work in them. And it also suggests that we should stop fooling ourselves into thinking that we can make drastic cuts to education funding and get better results.

- Matt Di Carlo