These authors wander aimlessly around a data wilderness, searching for positive evidence about school vouchers. Their report attempts to make the case that New York City partial vouchers of $1,400 per year to attend private elementary schools for three years had later positive impacts on college attendance, full-time college enrollment and attendance at selective colleges for African American students. It received lavish media attention, including a foot-stomping commentary by the report’s authors in the Wall Street Journal that scolds President Obama for what they regard as his outrageous failure to line up behind voucher policies.

To help understand the problems with this report, let’s all mentally travel to Sunnyside, Nevada, which hit a high temperature of only 14°F on January 17, 2012. Even while the world was experiencing record heat, Sunnyside posted a record cold for that date. If we wanted to distract attention from overall warming trends, we might lead with this and other cherry-picked data. It’s an old trick that often works, if nobody pays attention to the overall trends and if nobody questions the cherry-picking.

Yet this is essentially the approach used by the Bunkum-winning Brookings report, which finds positive college-related impacts for African American students (but not for other students) who had received vouchers back in elementary school. The researchers, of course, had no a priori reason to think that African Americans would benefit in this way from vouchers, when other students do not. They simply explored the data, found lots of results showing no voucher benefits and then found this one (akin to Sunnyside, Nevada) that helped support their advocacy of vouchers.

Our reviewer did not criticize the decision to explore; instead, she points out that the results of foraging through data should be presented as such: “Had Chingos and Peterson framed the finding for African Americans as an encouraging, exploratory hypothesis deserving of further testing, I would not have been alarmed by the report. But the study’s results absolutely do not merit headlines such as ‘Vouchers promote college attendance for African Americans.’”

The reason is simple. As our reviewer indicates, “Contrary to how it was presented, the main finding of this new report should be that, using a rigorous experimental design in which vouchers were randomly assigned to students, the estimated college enrollment rates of students with and without vouchers were not different from one another.”

Buried on p. 12 of the report is the statement that for the total sample, there was “a tiny insignificant impact.” As for the claims of a positive effect on college attendance of African Americans, there were no statistical differences between ethnic groups. Yet the authors chose to trumpet a positive effect for African Americans.

If there were indeed masked positive effects for one subgroup yearning to express themselves, there also must be some negative effects cruelly neutralizing the hidden voucher gains. That is, there must have been some students doing worse. Who are they? The authors refused to say. And, despite the prodding by our reviewer, they did not make public their full findings; they simply expected readers to trust them.

This sort of substandard research practice is why the Bunkum Award was born. The evidence in this study—honestly read—simply does not say what the authors said it does.