Bunkum Awards
The Bunkum Awards highlight nonsensical, confusing, and disingenuous education reports produced by think tanks. They are given each year by the Think Twice think tank review project to think tank reports judged to have most egregiously undermined informed discussion and sound policy making.
Bunkum Background
According to the MacMillan English Dictionary Magazine, the word bunkum became synonymous with "nonsense" around 1820. It was originally spelled "buncombe," after Buncombe County, North Carolina, home of Representative Felix Walker, who delivered a seemingly endless speech which many in his audience felt to be meaningless and irrelevant. Walker refused to stop talking, declaring himself to be determined to deliver a speech "for Buncombe." Thus, bunkum became a term for long-winded nonsense of the kind often seen in politics, and from there progressed to the more general meaning of just plain "nonsense."
Learn more about the history of the Bunkum Awards by reading these Education Week Commentaries:
Bunkum Awards
The ‘Magic Potion’ Award
This South Carolina Policy Council Education Foundation report, authored by Sven Larson, wins this year’s prize for the most fatuous cause-and-effect claim. The claim is that school choice will miraculously (our word, not theirs) decrease the unemployment problem of five poor, rural South Carolina counties. Reading this report, one feels transported to an old-time traveling medicine show peddling magic potions: That’s right, ladies and gents, just by being exposed to school choice programs, students will be given a dose of the elixir of private entrepreneurialism that will result in 329 student jobs in 123 new businesses!
As our reviewer explains, the claims in the report are based overwhelmingly on the “tuitioning” programs established in Vermont and Maine back in the 19th century, whereby very small towns that do not operate schools pay tuition to schools in other towns. Milwaukee’s choice adventures are also cited, as children there are said to be more entrepreneurial. But none of the source documents were peer-reviewed and the data are simply cross-sectional; no causal inferences are (sensibly) possible. Moreover, while ascribing New England small town characteristics to South Carolina counties may seem fanciful, it is not too far a reach for Mr. Larson. Unencumbered with traditional citations, the author announces that the benefits of vouchers are “widely documented.”
The ‘Remedial Arithmetic’ Award
The goal of this CATO Institute report was to calculate a comprehensive amount spent by K-12 public school districts. Yet, not being content with the actual legally required definitions and conventions used by trained and experienced finance specialists, researcher Adam Schaeffer calculated the “real” costs of public education using his own, extraordinarily creative formula. He found schools cost a lot more than he thought they should. In fact, his estimates of “real” costs vary from 3% to 151% of the government’s numbers.
The CATO breakthrough that we most honor with this award was the decision by Schaeffer to double-count expenditures by adding both capital construction and debt service to his calculation. As our reviewer explained, “most capital construction expenditures are not paid with taxpayer dollars. They are paid with proceeds from bonds. Taxpayer dollars then service the debt. This is key. The cost of a house bought with a loan is not the purchase price plus the cost of the loan. For a school district, what taxpayers are paying for in any year is debt service in that year.” Interestingly, Schaeffer then infers that it’s the National Center for Education Statistics method of cost-calculation that’s designed to mislead the public.
The ‘Plural of Anecdote is Not Data’ Award
Fittingly, we have a double winner in this plural category.
In Fix the City Schools, the Reason Foundation argues for "portfolio" school districts focused on closing low-performing schools and opening new ones under the management of autonomous corporations and sings Hosanna praises to the improvements in student achievement in New Orleans in the post-Katrina era. However, assuming real gains were made, there are myriad reasons unrelated to the portfolio approach that could explain some or all of them, including the massive exodus of low-income children from the city, plus a significant increase in resources. Miraculously, the findings from New Orleans, supplemented by examples from other cities, all evangelically testify to the healing powers of the portfolio school approach.
Not to be outdone, the Fordham Institute, in Charter School Autonomy,contends that charter schools have been deprived of the autonomy necessary for them to deliver on the innovative practices they promised. How could charters truly be innovative if they are still laboring under the yoke of bureaucratic control? It’s an interesting excuse for the not-particularly-impressive results of charter schools. But outside of anecdotes and rhetoric, there is nothing in this report that addresses autonomy in relation to financial performance, resource allocation, academic results, or other key school characteristics and outcomes. The authors simply fail to address their research question – whether and how authorizers’ constraints have had an adverse impact upon charter school autonomy and success.
Now, if we were being completely fair, this award would also be shared by the Blueprint research summaries published by the U. S. Department of Education. All six reviews noted the paucity of research evidence, and the department’s documents were replete with shadowed boxes providing war stories and anecdotes of “success.” In some cases, the boxed anecdotes were actually longer than the accompanying text. But in the “share the wealth” spirit of the administration, we decided that others should be allowed to bask in the glory of this Bunkum Award.
The ‘F Double Minus’ Award
There is a crowded industry devoted to rating and ranking schools and states. The cool part about these ratings is that the imperious think tank raters get to assign a grade of F Double Minus to states and schools that have not adopted their group’s cherished reforms.
The competition is especially plentiful and fierce in this category, but Herbert Walberg and Mark Oestreich of the Heartland Institute vanquished all contenders with their 2010 State School Report Card. Our honorees ranked states (and the District of Columbia) on student achievement, low education expenditures, and “adherence to learning standards,” as well as a ranking based on an average of the first three. They created the indices and validated them by a rigorous examination of their preconceived proclivities and then highlighted the top- and lowest-performing states for each of the indices. Although the report explains how the indices were devised, it does not cite any research or rationale to support the methods, other than the data sources themselves plus reports from “several think tanks.” As the report acknowledges, the researchers also don’t even try to control for state variations in demographics or other factors. The report then assigns letter grades to each of the states (plus DC), with a forced distribution: 10 states are assigned A's, B's, C's, and D's, and 11 states must get F's. The beauty of this normative system is that it guarantees it will always be raining – there will always be failing states. Even setting aside problems with the criteria chosen, this seems a bit problematic. Anyway, after acknowledging that there is “very little empirical evidence” as to why some states do better than others, the authors stroll along to their conclusion that education quality is poor and that school choice is the obvious remedy. Assuming there was an acceptable method in this pile of subjective accretion, it is completely unconnected with these final recommendations.
The runner up in this category is a report from the Education Trust, called Stuck Schools, which suggests a framework for identifying chronically low-performing schools in need of turnaround. Among four main problems identified by the reviewer was the following:the norm-referenced methodology guarantees "failed" schools independent of any true performance or improvement level by the school. Sound familiar?
The 'If I Say It Enough, Will It Still Be Untrue?' Award
While the Heritage Foundation receives this award for publishing Closing the Racial Achievement Gap, by Matthew Ladner and Lindsey Burke, it should really share this award with the think tanks in Colorado, New Mexico, Arizona, Wisconsin, Oklahoma, Utah, and Indiana, as well as the Hoover Institution and the Pacific Research Institute, all of which also published essentially the same report. Perhaps even the derivative articles by Dr. Ladner at National Review Online and Foxnews.com deserve some recognition.
But Ladner’s fecundity isn’t really what sets this work apart. It’s his willingness to smash through walls of basic research standards in his dogged pursuit of his policy agenda. In this case, the agenda is a passel of Florida reforms: vouchers funded by tax credits, charter schools, online education, performance-based teacher pay, test-score grading of schools and districts, test-based grade retention, and alternative teacher certification. He likes them. A lot. And he contends that other states should adopt the same package because, in his vision, they have clearly caused an improvement in Florida schools.
The problem, alas, is with the “caused” part. Nothing in the data or analyses of Dr. Ladner or the Heritage Foundation comes even close to allowing for a causal inference. Instead, the report offers uncontrolled descriptive data focused on NAEP fourth-grade reading scores. Those scores increased, so Ladner’s favored policies must be working. Our reviewer threw a few pails of cold water on that conclusion, noting first that other policies were also in effect during the time period studied. Florida has an excellent supportive reading program aimed at early grades; its accountability system strongly targets resources at lower-performing schools, and it has one of the nation’s most aggressive class-size reduction reforms. Ladner has steadfastly forsworn the possibility that these factors may have played a major role in any achievement improvements.
The reviewer also pointed out the reason why Ladner focused on fourth-grade reading and why NAEP growth in other subjects and grades wasn’t nearly as impressive: the researchers failed to properly address the state’s policy of retaining third-graders with low reading scores. This gave the bottom-scoring students another year to grow before they took the fourth-grade reading test – voilà. By analogy, consider growth in height instead of growth in test scores. If two states wanted to measure the average height of their fourth-graders, but one state (Florida) first identified the shortest 20% of third-graders and held them back to grow an additional year before measurement, the study’s results might be considered biased.
But Dr. Ladner has found his bone, and he’s not letting go. When confronted with the reviewer’s critique, he responded, "The change averse may wish to quibble over the details, or agonize over just what reform did how much of what," but the bottom line is that Florida’s fourth-grade scores increased dramatically. Indeed they did. And no research-bound egghead is going to mess up his good causal story.
The 'Good Enough for Government Work' Award
This year’s Grand Prize goes to Secretary Arne Duncan and his U.S. Department of Education staff for the exceptionally disappointing low quality of their research reviews supporting their plans for the reauthorization of ESEA (aka, the Blueprint). Our esteemed panel of judges solemnly considered whether the federal government was even eligible for such an award. With so many resources at its disposal, the government seems to have an unfair advantage. But the Blueprint research summaries stood out in two ways that we felt needed recognition. First, they almost religiously avoided acknowledging or using the large body of high-quality research that the federal government itself had commissioned and published over the years. Second, they first raised our expectations with repeated assurances that recommended policies would be solidly grounded in research – only to then dash those hopes in research summary after research summary.
The issues addressed in the Blueprint and the research summaries are certainly vital to the nation’s education system – standards, teacher quality, comprehensive education, special needs, safe and healthy students, and charter schools. But across the board, our reviewers found the work to be of inadequate quality. One reviewer was astounded that the administration did not mount a comprehensive defense of its central education policies. The research summary reviewed by another was described as a “political text that starts with a conclusion and then finds evidence to support it.” Then, there was the question of critical omissions – such as the complete absence of a rationale for the chosen accountability system and intervention models. In terms of sources, the research summaries were often found to be summarizing non-research. For example, only 10% of the 80 or so citations in the “Great Teachers, Great Leaders” summary referred to peer-reviewed research sources.
The Annual Friedman Foundation Johnny-One-Note Award
The annual Friedman Foundation Johnny-One-Note Award for promoting an educational cure-all through the miracle of cloning goes to (drum roll please) the Friedman Foundation.
The Foundation has, over the past three years, cloned the same study on the cost of drop-outs in at least seven states, a tax credit voucher report in at least six states, and opinion polls on school choice in 15 states. Amazingly, all these reports lead to the same conclusion: vouchers and other forms of school choice will save money and improve student outcomes. (Given the miraculous power Friedman assigns to vouchers one might be forgiven for wondering if implementing voucher programs would take off unwanted weight and leave partners fully satisfied as well.) The basic technique used by Friedman researchers is to take the same report, change the name of the state, plug in some state-specific data, vary the title a bit, and come up with the predetermined conclusion.
We also should not fail to acknowledge a non-clone Friedman Foundation offering that we reviewed in 2009. The Foundation’s Win-Win report argues that vouchers help both the private and the public schools. It purports to gather all available evidence on the competitive effects of vouchers and is able to find only seventeen studies, most of which were produced by voucher advocacy organizations. From this thin and biased review, the report concludes that there is a consensus on the matter. In truth, existing research provides little reliable information about the competitive effects of vouchers, and this report does little to help answer the question.
The Innovations in Promoting Alternative Teacher Certification Award
Given the great interest in alternative teacher and administrator preparation programs, studies such as this prize winner tend to attract considerable attention. And readers had reason to expect high quality from a federally funded study conducted by Mathematica Policy Research, a respected research organization.
The researchers, using a random assignment design, reported “no evidence” that traditionally trained teachers provided better student scores than non-traditional or alternatively trained teachers. There were no caveats in the announcement of the study’s conclusions. But our third party expert reviewers explained that there should, in fact, have been many, many caveats. Here’s just a taste: small sample size focused overwhelmingly on urban, poor, heavily minority, early grade students; troubling sampling methods; and a failure to distinguish the “treatments” that alternative certification and traditional certification teachers provided (meaning that members of the two compared groups were both undertrained and had substantially overlapping preparation experiences).
The reviewers stressed that the study’s primary limitations are due to the fact that it intentionally sampled from a unique subset of schools: those that routinely hire alternatively certified teachers. Since the study necessarily matched alternative-certified and traditionally-certified teachers working at the same hard-to-staff schools, it is quite likely that the traditionally certified teachers who made up the comparison group in this study were substantially less qualified than the average traditionally certified teacher.
The review documents additional problems as well. But what’s interesting, and particularly telling, is that even with the deck stacked strongly against traditional teacher preparation, the study included many analyses that found traditionally trained teachers outperformed their alternative route counterparts. It’s just that the report’s authors chose not to fully report and acknowledge these findings in the report’s conclusions.
The Misdirection Award: Keep our Eyes off What Works
This electronically published book, published by the Hoover Institution and authored by Fordham Foundation president and Hoover Senior Fellow Checker Finn, joins the ranks of our dubious honorees. Misdirecting readers from a mountain of empirical, peer reviewed and widely accepted evidence, Finn cherry-picks a few weak studies to criticize proposals for universal preschool. Our third-party expert reviewer summed up Finn’s work as “errors, exaggerations, misrepresentation and logical inconsistency.” Among the reviewer’s catalog of fourteen major errors, he notes that actual costs are exaggerated by a factor of two while immediate and long-term well-documented effects are under-reported or not reported accurately. The book also ignores numerous meta-analyses of preschool research and cost-benefit studies that have found a clear social and financial benefit for early education, amounting to hundreds of thousands of dollars per child.
Finn also praises the Florida program as a model even though there are no accountability data available from that program, and the program is structurally inequitable. In the push for a non-universal program, “The book proposes a vague, targeted alternative that is entirely fictional, but which, like the mythical gryphon, is especially powerful and majestic.”
The Data Dodger Award
This report, published by the National Bureau of Economic Research and lead authored by Hoover Institution Senior Fellow Caroline Hoxby, initially escaped our attention. But the Washington Post and many charter advocates trumpeted the findings so loudly that we really had no choice but to seek a review. We were glad we did. In a revolutionary reversal of research procedures, Hoxby and her colleagues announced her results to the media and policymakers while withholding much of the actual information on which her results were based. Hers was a breakthrough in scientific methodology and a tribute to the non-accountability of pro-accountability researchers.
As for the report itself, Hoxby claimed that charter schools in New York City worked better than public schools, pointing out that she had taken advantage of a natural experimental design as students were assigned by lottery. However, the expert third party review notes several likely sources of bias which probably resulted in Hoxby’s inflated finding. Among these, the study relies on statistical models that include 3rd grade test scores, measured after the admission lotteries had taken place. Because of that timing, those scores could be affected by whether students attend a charter school, meaning that Hoxby’s chosen statistical models destroy the benefits of the very randomization that she and her supporters rely on as the main strength of the study’s design. Of course, the reviewer couldn’t quantify the extent of the overestimate since Hoxby had left out the information that would be needed for readers to engage in such an independent review and analysis. Trust, and don’t verify appears to be the operational accountability philosophy.
Nevertheless, accompanied by a formidable cloud of statistical formuli and dressed up in elaborate theoretical assumptions and explanations, this work was widely heralded as proving the policy wisdom of the charter school reform. As our reviewer pointed out, New York City’s charter schools might genuinely be improving student outcomes; however, this study—because of the information it withheld and its methodological shortcomings—does not and cannot resolve the issue.
The Time Machine Award
This Reason Foundation report has multiple features that make it an award winner. It engages in definitional acrobatics, pouring a kitchen sink’s worth of assorted reforms into a vessel it calls Weighted Student Formula (WSF) reforms. And, in a truly breathtaking innovation, the report enters its time machine and attributes positive reform outcomes to policy changes that had not yet been implemented. In broad terms, WSF reforms involve linking funding to each student, with that funding calculated as the student’s base allocation and any additional funds for special needs, economic deprivation or other reasons. The Reason report somehow manages to squeeze into this WSF concept three additional reforms: (a) site-based management; (b) site-based budgeting; and (c) school choice. The expert third party reviewer said this about the Reason “umbrella labeled as WSF:” “[it] deceptively suggests that all related policies are necessarily good—even going so far as to credit those policies for improvements that took place before the policies were implemented.”
“The report then irresponsibly recommends untested, cherry picked policy elements, some of which may substantially undermine equity for children in the highest-need schools within major urban districts.” For example, the plan suggests that extra funds for economically deprived students be eliminated but that added money should be given to gifted and talented students. The report also ignores a large body of relevant literature on within-district equity and school site management in its uncritical effort to find support for the foundation’s ideological policy preferences.
The Consolation Prize
In this third year of the Bunkum awards, the true barons of bunkum have bullied their way to the front. Perennial powerhouses Friedman and Fordham once again made the list for poor research and execution. Sadly, the Cato Institute in 2008 failed to hit its mark. Their entry into the competition was a global review called Markets vs. Monopolies in Education: A Global Review of the Literature. Normally, a report that excludes major studies, ignores selection bias, and oversimplifies the complex characteristics of educational markets would be a contender. Further, in an act of self-aggrandizing puffery, the report proclaims itself to be of “profound” importance to U. S. educational policy. Alas, they were knocked from this year’s rankings by their application of alleged lessons from the educational systems of Pakistan, India, Tanzania, Ghana and Nigeria to the fundamentally and structurally different system of the United States. This left our judges no choice but to dismiss the report as merely silly.
The Maybe It’ll Be True If We Say It One More Time Award
The Friedman Foundation garners the MIBTIWSIOMT award this year for building two distinct franchises on little more than fixated false claims.
One set of five cloned state studies offered the repetitive conclusion that high school dropouts would be reduced and economic prosperity advanced if voucher programs were introduced in each state. Although severely handicapped by the fact that there is no significant evidence that vouchers will reduce drop-outs, the author hangs his hat on a single 1998 study. Ironically, the Friedman reports themselves criticize the very approach used in this cherry-picked article: reliance on administrative counts (such as reports of school principals) to estimate high school graduation.
Even more impressive than this first effort, the Foundation stamped out ten separate state surveys, all of which came to the conclusion that potential voters in each state endorsed private school vouchers. Admittedly, the surveys suffered from biased questions and were administered to respondents whose own responses showed limited knowledge of the educational policy issues they provided opinions about. But the important thing is that these respondents’ conclusions were substantially more pro-voucher than the Kappan Gallup poll responses on the same topic (except that the Friedman survey reports never mentioned the Gallup research). Perhaps most damning, the authors reached so far that their pro-voucher conclusions were not even supported by their own problematic survey data.
The Rose Colored Blinders Award
This award is bestowed upon Public Charter Schools: A Great Value for Ohio’s Public Education System, authored by Matthew Carr and Beth Lear and published by Ohio’s Buckeye Institute. While many advocacy think tanks annually contend for this high honor, it was earned this year by the Buckeye Institute for its determined disregard of an extensive, non-partisan, and quite relevant state official report, its non-existent literature review, and its decisive failure to comprehend the state funding formula lying at the heart of its analysis. Our reviewer found that these bundled blunders lead the authors to the baseless conclusion that each charter student saves as much as $4,030 for the host public school district. He described the report as “unfounded and outlandish,” as well as “ridiculously false, deceitful, and patently misrepresent[ing] how the funding of public schools works.”
The Charles Murray Prize for Identifying Who Shouldn’t be Educated
The Fordham Institute and the Brookings Institution jointly win the Murray for their articles authored by Tom Loveless. His back-to-back winners came only three months apart but both showed his enduring commitment to convincing policy makers that too much effort is spent academically challenging the wrong children.
High Achieving Students in an Era of NCLB attempts to build the case that concentrating on low-achieving children diminishes the growth of the higher achievers, who accordingly become “languid.” Dr. Loveless comes to this conclusion by presenting NAEP score comparisons of trends among high- and low-scorers, showing faster growth at the bottom of the distribution. “[T]his trend,” the report concludes, “suggests a missed opportunity to promote achievement among high achievers.” The Think Tank Project reviewer, however, pointed out a troubling inconsistency. The report correctly notes that its correlational analyses can not be used to draw causal inferences, but it also makes patent causal inferences to bolster its policy recommendations. This over-reach is most apparent in the foreword by Fordham's Chester Finn and Michael Petrilli.
Loveless returns to the ‘wasted energies’ theme in The Misplaced Math Student: Lost in Eighth-Grade Algebra, published by the Brookings Institution. In this piece, Loveless contends that having low-achieving students in algebra classes with highly proficient students dampens opportunities for the best students and dooms many lower achievers to failure. Only one peer reviewed article is discussed, and it comes to a different conclusion – so Loveless criticizes it for selection bias. His research method is to use state NAEP scores and correlate them with algebra-taking rates in each state. Finding no relationship he concludes that his hypothesis is sustained, despite acknowledging that his correlational findings should not be used to argue that causal relationships have been found. Sound familiar?
Chutzpah Award
We give this special award to the Friedman Foundation, which places on page two of each of its reports a section entitled, "Our Challenge To You." The passage begins, "Our research adheres to the highest standards of scientific rigor," and it ends with "prove us wrong. Judge our work by scientific standards and see how it measures up. If you can find anything in our work that doesn’t follow sound empirical methods, by all means say so. We welcome any and all scientific critique of our work..." Over the first two years of the Think Tank Review Project, six reviews of Friedman studies were published. Two reviews included substantial praise, mixed with varying degrees of criticism; the four others generally described the Friedman reports as misleading and poorly grounded – certainly not adhering to "the highest standards of scientific rigor." Yet, notwithstanding the page two "Challenge," we have received no communications from the Friedman Foundation to date, and the key shortcomings found by reviewers have never been addressed by the Friedman Foundation. The Foundation has also not softened its penchant for publishing reports that selectively and misleadingly cite research and that routinely make grand and unsupported inferential leaps to arrive at policy recommendations arguing for privatization.
With Friends Like These Award
Ohio’s Buckeye Institute, in Shortchanging Disadvantaged Students: An Analysis of Intra-district Spending Patterns in Ohio, argued against increased state funding for school districts on the grounds that those districts did not fairly allocate money to schools serving disadvantaged students. The review of the report raised serious questions about the validity of the calculations and conclusions. But beyond that, we were somewhat taken aback by the report’s novel argument which was essentially, 'Because we care about disadvantaged students getting insufficient resources, we recommend against more state funding.'
Who Reads Warning Labels? Award
In How Much Are Public School Teachers Paid the Manhattan Institute author used hourly earnings data to contend that teachers are better paid than most white-collar professionals. This might have been impressive except, as our reviewer noted "this approach is fundamentally flawed because the [dataset’s] calculation of weeks and hours worked is very different for teachers and other professionals. In fact, the Bureau of Labor Statistics – which publishes the [dataset] – has explicitly warned its users not to use hourly rates of pay in this exact same context."
60 Cent Solution Award
A May 2007 Friedman report trumpeted its finding that the nation’s "twelve [voucher] programs have saved a total of nearly half a billion dollars" (School Choice by the Numbers: The Fiscal Effect of School Choice Programs 1990 – 2006). Our reviewer, however, noted that even if the report’s flawed calculations were accepted, a savings of a half-billion dollars was "a savings of less than 1/100th of one percent of annual public school spending, or about 60 cents per child per year."
Inferential Long Jump Award
Unquestioned faith that privatization in one form or another will solve educational problems was a pervasive think tank theme throughout 2007. Repeatedly, our reviewers pointed out instances where reports made an Olympian inferential leap to market remedies, even though the rest of the report provided little or no empirical support for that conclusion. Our two winners bounded the longest distance: a Cato report entitled, End It, Don’t Mend It: What to Do with No Child Left Behind and the Report Card on American Education, published by the American Legislative Exchange Council.
