Review says analyses rely on unreliable and poorly grounded assumptions and that alternative scenarios could have offered greater insight
Contact: Jennifer Jellison Holme, (512) 475-9398; (email) jholme@mail.utexas.edu
Kevin Welner, (303) 492-8370; (email) kevin.welner@gmail.com
TEMPE, Ariz and BOULDER, Colo. (Jan. 27, 2009) -- A pair of recent reports largely dismiss the value of inter-district choice--enrollment across district lines--for students in schools labeled as failing. The two reports' analyses and conclusions were, however, often based on a series of unreliable and poorly grounded assumptions, a new review finds.
The reports, Plotting School Choice: The Challenges of Crossing School District Lines, released in August 2008, and In Need of Improvement: Revising NCLB's School Choice Provision, released in November 2008, were reviewed for the Think Tank Review Project by Jennifer Jellison Holme and Meredith P. Richards. Holme is an assistant professor of Educational Policy and Planning at the University of Texas at Austin, where Richards is a doctoral student.
Plotting School Choice and In Need of Improvement were both written by Erin Dillon, a policy analyst for Education Sector, a Washington, D.C.-based education think tank, which published the reports.
Under the No Child Left Behind Act (NCLB), students whose schools fail to make Adequate Yearly Progress for two years in a row are eligible to transfer to a non-failing public school in their district. Many fewer do so than had been originally expected by the law's supporters. To increase choice options, policy makers and advocacy groups have proposed amending the legislation to also permit those students to transfer to school outside their home districts.
Dillon's reports are largely skeptical of that proposal. Both use computer-software mapping (GIS) technology to estimate drive-time distances between all lower-performing and all higher-performing schools in each grade level for a sample of states. The first report looks at three states: California, Florida, and Texas. The report presents analyses of that information under a series of specific assumptions to estimate the potential impact of allowing students in low-performing schools to choose schools outside their home district. The resulting findings estimate that 10 to 20 percent of students would benefit from inter-district choice, while 80 to 90 percent would remain in the same, low-performing schools.
The second report revisits California and also looks at the Chicago Public School District. It also alters statistical assumptions from the August report. While Dillon concludes that in Chicago, expanding NCLB's choice provisions to include inter-district choice "is unlikely to benefit most students," the new California analysis is more optimistic about the prospects of benefits from inter-district choice.
Reviewers Holme and Richards credit the reports with bringing the potentially useful mapping technique to the discussion. They also credit Dillon with making the "important and well supported" point that, if the choice policies are to expand opportunity for the least-advantaged students, they must be carefully targeted to the most needy, providing them with information and transportation, and providing suburban districts with financial incentives to participate. Deregulated, open-enrollment policies tend to primarily benefit students already relatively advantaged.
After noting some concerns about the reports' lack of sufficient data to support the key findings, Holme and Richards explain the key problem: that the reports use unreliable and poorly grounded assumptions. The reviewers acknowledge that any analysis of these issues must be based on assumptions, but both reports would have been far stronger had the authors analyzed a wide range of scenarios, making and then testing various alternative, plausible assumptions. By doing so, the reports could have been made useful to policy makers beyond those few who might agree with the reports' current, specific assumptions.
• The reports each make different assumptions, many of which are not empirically grounded, about the number of students who would be eligible to exercise inter-district choice options. The reports would have been more useful had a range of student eligibility scenarios been presented.
• The reports limit the options available to choice-eligible students by assuming that receiving schools have only a 10% additional capacity, notwithstanding wide variance in actual schools' capacity for increased enrollment.
• The reports assume students or families would be willing to travel no more than 20 minutes. Yet a Boston inter-district choice program, for example, draws students willing to travel as much as three hours round-trip daily. The reviewers suggest the report should have analyzed various travel time assumptions.
Finally, Holme and Richards note, the reports do not adequately account for the significant variation in district sizes. Average district size varies greatly by region of the country, with smaller districts in the urban northeast and Midwest and larger, often county-wide, districts in the South and West. While larger districts are more likely to be able to offer students meaningful within-district choice, smaller ones are less likely to be able to do so, making inter-district choice more important for students in those districts.
"But by including only a limited range of (often poorly supported) assumptions, with little testing of alternative assumptions, and by failing to systematically consider nuances across contexts, the two Education Sector reports fail to offer policy makers useful guidance," the reviewers conclude.
Find Jennifer Holme's and Meredith Richards's review on the web at:
http://epicpolicy.org/thinktank/review-plotting-school-choice
CONTACT:
Jennifer Jellison Holme, Assistant Professor
University of Texas at Austin
(512) 475-9398
jholme@mail.utexas.edu
Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@gmail.com
About the Think Tank Review Project
The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.
Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."
**********
###
The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.
Visit EPIC and EPRU at http://www.educationanalysis.org/
EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).
###
**********