I read a recent blog from two researchers who assert that principals can improve students’ test scores. The researchers cite studies that support their claim (see below). These researchers received a large grant from the Wallace Foundation to alter their principal preparation program to turn out principals who can, indeed, raise students’ academic achievement.

I was intrigued by this post because as a district superintendent I believed the same thing and urged the 35 elementary and secondary principals I supervised—we met face-to-face twice a year to go over their annual goals and outcomes and I spent a morning or afternoon at the school at least once a year—to be instructional leaders and thereby raise test scores. Over the course of seven years, however, I saw how complex the process of leading a school is, the variation in principals’ performance, and the multiple roles that principals play in his or her school to engineer gains on state tests (see here and here). And I began to see clearly what a principal can and cannot do. Those memories came back to me as I read this post.

First the key parts of the post:

A commonly cited statistic in education leadership circles is that 25 percent of a school’s impact on student achievement can be explained by the principal, which is encouraging for those of us who work in principal preparation, and intuitive to the many educators who’ve experienced the power of an effective leader. It lacks nuance, however, and has gotten us thinking about the state of education-leadership research—what do we know with confidence, what do we have good intuitions (but insufficient evidence) about, and what are we completely in the dark on? ….

Quantifying a school leader’s impact is analytically challenging. How should principal effects be separated from teacher effects, for instance? Some teachers are high-performing, regardless of who leads their school, but effective principals hire the right people into the right grade levels and offer them the right supports to propel them to success.

Another issue relates to timing: Is the impact of great principals observed right away, or does it take several years for principals to grapple with the legacy they’ve inherited—the teaching faculty, the school facilities, the curriculum and textbooks, historical budget priorities, and so on. Furthermore, what’s the right comparison group to determine a principal’s unique impact? It seems crucial to account for differences in school and neighborhood environments—such as by comparing different principals who led the same school at different time points—but if there hasn’t been principal turnover in a long time, and there aren’t similar schools against which to make a comparison, this approach hits a wall.

Grissom, Kalogrides, and Loeb carefully document the trade-offs inherent in the many approaches to calculating a principal’s impact, concluding that the window of potential effect sizes ranges from .03 to .18 standard deviations. That work mirrors the conclusions of Branch, Hanushek, and Rivkin, who estimate that principal impacts range from .05 to .21 standard deviations (in other words, four to 16 percentile points in student achievement).

Our best estimates of principal impacts, therefore, are either really small or really large, depending on the model chosen. The takeaway? Yes, principals matter—but we still have a long way to go to before we can confidently quantify just how much.

I thoroughly agree with the researchers’ last sentence. But I did have problems with these assertions supported by two studies they listed.

*That principals are responsible for 25 percent of student gains on test scores (teachers, the report account for an additional 33 percent of those higher test scores). I traced back the source they cited and found these statements:

A 2009 study by New Leaders for New Schools found that more than half of a school’s impact on student gains can be attributed to both principal and teacher effectiveness – with principals accounting for 25 percent and teachers 33 percent of the effect.

The report noted that schools making significant progress are often led by a principal whose role has been radically re-imagined. Not only is the principal attuned to classroom learning, but he or she is also able to create a climate of hard work and success while managing the vital human-capital pipeline.

These researchers do cite studies that support their points about principals and student achievement but cannot find the exact study that found the 25 percent that principals account for in student test scores. Moreover, they omit  studies that show  higher education programs preparing principals who have made a difference in their graduates raising student test scores (see here).

I applaud these researchers on their efforts to improve the university training that principals receive but there is a huge “black box” of unknowns that explain how principals can account for improved student achievement. Opening that “black box” has been attempted in various studies that Jane David and I looked at a few years ago in Cutting through the Hype.

The research we reviewed on stable gains in test scores across many different approaches to school improvement all clearly points to the principal as the catalyst for instructional improvement. But being a catalyst does not identify which specific actions influence what teachers do or translate into improvements in teaching and student achievement.

Researchers find that what matters most is the context or climate in which the actions occurs. For example, classroom visits, often called “walk-throughs,” are a popular vehicle for principals to observe what teachers are doing. Principals might walk into classrooms with a required checklist designed by the district and check off items, an approach likely to misfire. Or the principal might have a short list of expected classroom practices created or adopted in collaboration with teachers in the context of specific school goals for achievement. The latter signals a context characterized by collaboration and trust within which an action by the principal is more likely to be influential than in a context of mistrust and fear.

So research does not point to specific sure-fire actions that instructional leaders can take to change teacher behavior and student learning. Instead, what’s clear from studies of schools that do improve is that a cluster of factors account for the change.

Over the past forty years, factors associated with raising a school’s academic profile include: teachers’ consistent focus on academic standards and frequent assessment of student learning, a serious school-wide climate toward learning, district support, and parental participation. Recent research also points to the importance of mobilizing teachers and the community to move in the same direction, building trust among all the players, and especially creating working conditions that support teacher collaboration and professional development.

In short, a principal’s instructional leadership combines both direct actions such as observing and evaluating teachers, and indirect actions, such as creating school conditions that foster improvements in teaching and learning. How principals do this varies from school to school–particularly between elementary and secondary schools, given their considerable differences in size, teacher peparation, daily schedule, and in students’ plans for their future. Yes, keeping their eyes on instruction can contribute to stronger instruction; and, yes, even higher test scores. But close monitoring of instruction can only contribute to, not ensure, such improvement.

Moreover, learning to carry out this role as well as all the other duties of the job takes time and experience. Both of these are in short supply, especially in urban districts where principal turnover rates are high.

I am sure these university researchers are familiar with this literature. I wish them well in their efforts to pin down what principals do that account for test score improvement and incorporate that in a program that has effects on what their graduates do as principals in the schools they lead.