Pro-voucher Polls Plagued by Biased Questions and Sampling Problems, Review Finds

Contact: Jon Lorence, (713) 743-3959; (email) jlorence@uh.edu
A. Gary Dworkin, (713) 743-3955; (email) gdworkin@uh.edu
Kevin Welner, kevin.welner@gmail.com

TEMPE, Ariz. and BOULDER, Colo. (May 22, 2009) -- A new report issued this week by the Friedman Foundation is part of a series of reports based on public opinion polls, each of which relies on polls containing poorly worded questions and suffering from problems in sampling. These problems were documented six months ago, in a study of the first 10 of these Friedman reports.

The earlier reports were reviewed for the Think Tank Review Project by professors Jon Lorence and A. Gary Dworkin, both of the University of Houston.

The 10 reports conclude that a majority of likely voters view public schools as performing unsatisfactorily, that respondents prefer private over public schools and want more educational choices, that vouchers should be available, and that potential voters are more likely to support candidates who back school choice proposals.

Lorence and Dworkin note that the reports fail to cite other randomized surveys, or other research literature, regarding opinions on vouchers or on public, private, or charter schools. By comparison, the reviewers cite a series of high-quality Gallup polls, conducted annually for Phi Delta Kappa, that include questions measuring public support for private-school vouchers and for public education in general. Those surveys have shown moderate fluctuation in their findings, but consistently find lesser support for voucher proposals than shown by the Friedman surveys. In fact, the Gallup surveys consistently show that more Americans oppose vouchers than support them.

The reviewers help readers understand why the Friedman surveys might yield these more favorable results for school choice proposals. They identify two primary problems.

First, they note that that the population samples surveyed might not represent the voting populations of their states nor the population of parents for whose children school vouchers are intended. Information to establish sampling accuracy is largely missing from the reports, and in at least one of the three instances where sufficient data were provided, the low response rate raises a red flag about possible sample bias. In the case of Ohio, the response rate was only 37%, lower than two of the three surveys discussed in the earlier review.

Additionally, Lorence and Dworkin found repeated instances in which the wording of questions appeared likely to bias the results. These questions appeared to be worded in ways to encourage respondents to favor private-school funding. The reviewers use the corresponding Gallup questions to highlight these problems with wording. Additionally, the survey questions in some instances used terms not widely understood, such as questions about proposals to grant tax credits to companies that finance private school scholarships (vouchers).

"Had the respondents been presented more neutral questions about tax credits and vouchers, the findings may have been less favorable towards these issues," the reviewers write.

Lorence and Dworkin observe that the 10 reports reviewed are aimed at driving a pro-school-choice agenda through the promotion of tax credits, vouchers, and charter schools. But in addition to problems with bias in the surveys themselves, they point out, there's a more fundamental problem: namely, that "while the responses of those surveyed in the 10 states may reflect their beliefs endorsing alternatives to public schools, their views should not be construed -- as they often are in these reports -- to mean that adopting vouchers and other pro-choice school policies will in fact improve the quality of education."

In short, Lorence and Dworkin conclude, the reports should not guide policy.

Find the review by Jon Lorence and A. Gary Dworkin on the web at:
http://epicpolicy.org/thinktank/review-voucher-surveys

Find the Friedman Foundation's Ohio report at:
http://www.friedmanfoundation.org/downloadFile.do?id=371

CONTACT:
Jon Lorence
University of Houston
(713) 743-3959
jlorence@uh.edu

A. Gary Dworkin
University of Houston
(713) 743-3955
gdworkin@uh.edu

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
kevin.welner@gmail.com

About the Think Tank Review Project

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."

**********
###

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********