Given the great interest in alternative teacher and administrator preparation programs, studies such as this prize winner tend to attract considerable attention. And readers had reason to expect high quality from a federally funded study conducted by Mathematica Policy Research, a respected research organization.

The researchers, using a random assignment design, reported “no evidence” that traditionally trained teachers provided better student scores than non-traditional or alternatively trained teachers. There were no caveats in the announcement of the study’s conclusions. But our third party expert reviewers explained that there should, in fact, have been many, many caveats. Here’s just a taste: small sample size focused overwhelmingly on urban, poor, heavily minority, early grade students; troubling sampling methods; and a failure to distinguish the “treatments” that alternative certification and traditional certification teachers provided (meaning that members of the two compared groups were both undertrained and had substantially overlapping preparation experiences).

The reviewers stressed that the study’s primary limitations are due to the fact that it intentionally sampled from a unique subset of schools: those that routinely hire alternatively certified teachers. Since the study necessarily matched alternative-certified and traditionally-certified teachers working at the same hard-to-staff schools, it is quite likely that the traditionally certified teachers who made up the comparison group in this study were substantially less qualified than the average traditionally certified teacher.

The review documents additional problems as well. But what’s interesting, and particularly telling, is that even with the deck stacked strongly against traditional teacher preparation, the study included many analyses that found traditionally trained teachers outperformed their alternative route counterparts. It’s just that the report’s authors chose not to fully report and acknowledge these findings in the report’s conclusions.