Contact:
Matthias von Davier, ETS
(609) 734-1717
mvondavier@ets.org

William Mathis, NEPC
(802) 383-0058
William.Mathis@colorado.edu

BOULDER, CO (March 31, 2011) – Cross-Country Evidence on Teacher Performance Pay, a report recently published by the Program on Education Policy and Governance (PEPG) at Harvard, claims that students in nations where teachers are paid on “merit” score higher on an international achievement test. Matthias von Davier, principal research scientist at the Education Testing Service, reviewed the PEPG report for the Think Twice think tank review project. Von Davier found that the analyses and interpretations offered (also published in an abridged form in the journal Education Next) fail to heed the report’s own cautions, and it has too many crucial caveats to provide any useful policy guidance.

The review is published by the National Education Policy Center, housed at the University of Colorado at Boulder School of Education.

The report, authored by Ludger Woessmann of the University of Munich, contends that in those nations where teachers are paid based on student achievement or another performance-based measure, students have higher reading and math scores on the Programme for International Student Assessment (PISA) achievement test, an international exam given to 15-year-olds.

Woessmann’s findings rest on a series of regression analyses of PISA scores. The data are analyzed at the country level. Because there are only 28 data points (nations) in the analysis, the small sample size “requires extreme caution in interpretation,” von Davier writes. Simply excluding or including a single country in the analysis “results in large shifts in the size of the reported relationships.” That is, the presence of performance pay may or may not have an effect on PISA scores, but this sort of study can do little to answer that question.

Von Davier notes that, “Although the author lists numerous caveats, his broad conclusions do not heed these cautions.” The caveats are crucial to understanding the usefulness of the study’s findings. For instance, differences in the way various countries provide so-called performance-based pay are not properly considered. Perhaps one type of approach is beneficial, while another is detrimental. Additionally, variations in the length of time one or another country has employed such a pay system are not addressed. A system that hasn’t been in place long enough to have a strong effect is treated equally to one that has been in place much longer.

Von Davier points out that differences among nations, “could be due to any number of factors” – certainly there are enormous differences between countries and between their educational systems. So teasing out differences in PISA scores and then attributing those differences to the presence or absence of a merit-pay system is problematic.

As a result of the multiple limitations in the report’s data and of issues in the analytic methods used, von Davier concludes, that more careful, in-depth studies of differences between educational systems are needed before looking to international comparisons to substantially inform our own decisions about performance pay.

Find Matthias von Davier’s review on the NEPC website at:
http://nepc.colorado.edu/thinktank/review-pisa-performance-pay

Find Cross-Country Evidence on Teacher Performance Pay, by Ludger Woessmann on the web at:
http://www.hks.harvard.edu/pepg/MeritPayPapers/Woessmann_10-11.pdf.

The Think Twice think tank review project (http://thinktankreview.org), a project of the National Education Policy Center, provides the public, policy makers, and the press with timely, academically sound, reviews of selected publications. The project is made possible in part by the support of the Great Lakes Center for Education Research and Practice.

The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on NEPC, please visit http://nepc.colorado.edu/.

This review is also found on the GLC website at http://www.greatlakescenter.org/