Pro-voucher Polls Plagued by Biased Questions and Sampling Problems, Review Finds

Contact:
Kevin Welner, kevin.welner@colorado.edu

TEMPE, Ariz. and BOULDER, Colo. (November 16, 2009) -- A new report issued today by the Friedman Foundation and a collection of Virginia and national organizations is part of a series of reports based on public opinion polls, each of which relies on polls containing poorly worded questions and suffering from problems in sampling. These problems were documented back in December of 2008, in a study of the first 10 of these reports. The Virginia report is now the 15th in this series.

The earlier reports were reviewed for the Think Tank Review Project by professors Jon Lorence and A. Gary Dworkin, both of the University of Houston.

The 10 reports conclude that respondents prefer private over public schools and want more educational choices, that vouchers should be available, that potential voters are more likely to support candidates who back school choice proposals, and that a majority of likely voters view public schools as performing unsatisfactorily. While Virginia respondents tended to have a more positive view of public schools than as reported for the other states, the basic pattern is repeated in this new report.

Lorence and Dworkin note that the reports fail to cite other randomized surveys, or other research literature, regarding opinions on vouchers or on public, private, or charter schools. By comparison, the reviewers cite a series of high-quality Gallup polls, conducted annually for Phi Delta Kappa, that include questions measuring public support for private-school vouchers and for public education in general. Those surveys have shown moderate fluctuation in their findings, but consistently find lesser support for voucher proposals than shown by the Friedman surveys. In fact, the Gallup surveys consistently show that more Americans oppose vouchers than support them.

The reviewers help readers understand why the Friedman surveys might yield these more favorable results for school choice proposals. They identify two primary problems.

First, they note that that the population samples surveyed might represent neither the voting populations of their states nor the population of parents for whose children school vouchers are intended. Information to establish sampling accuracy is largely missing from the reports, and in one instance (a Montana survey) where sufficient data were provided, the low response rate raised a red flag about possible sample bias. This same problem arises here: the author of the Virginia report claims a response rate of only 26%. As Lorence and Dworkin explained in their original review, such a low response rate “considerably lessens the likelihood that the study’s findings are representative of the state’s voting population.”

Additionally, Lorence and Dworkin found repeated instances in which the wording of questions appeared likely to bias the results. These questions appeared to be worded in ways to encourage respondents to favor private-school funding. The reviewers use the corresponding Gallup questions to highlight these problems with wording.

"Had the respondents been presented more neutral questions about ... vouchers, the findings may have been less favorable towards these issues," the reviewers write.

Lorence and Dworkin observe that the 10 reports reviewed are aimed at driving a pro-school-choice agenda through the promotion of tax credits, vouchers, and charter schools. But in addition to problems with bias in the surveys themselves, they point out, there's a more fundamental problem: namely, that "while the responses of those surveyed in the 10 states may reflect their beliefs endorsing alternatives to public schools, their views should not be construed -- as they often are in these reports -- to mean that adopting vouchers and other pro-choice school policies will in fact improve the quality of education."

In short, Lorence and Dworkin conclude, the reports should not guide policy.

Find the review by Jon Lorence and A. Gary Dworkin on the web at:
http://epicpolicy.org/thinktank/review-voucher-surveys

Find the Virginia report at:
http://www.friedmanfoundation.org/downloadFile.do?id=396

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
kevin.welner@colorado.edu

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********