In 2021, a team of researchers published a systematic review of 20 years of research on education leadership and its connection to student learning. It won a significant award from the leading research professional association in education, and it remains one of the most popular resources on the Wallace website. The authors of How Principals Affect Students and Schools, Jason Grissom, Anna Egalité, and Constance Lindsay, analyzed more than 200 studies to detail how principal practices have positive impacts on their schools.
The team also broke ground in finding connections between an effective leader and student achievement. They conducted a meta-analysis (a “study of studies”) of six studies that had all used the same methodological approach in their work. These six studies had each applied value-added modeling, or VAM—commonly used to measure a teacher’s direct impact on student test scores—to estimate the indirect effects that principals have on student achievement. The results of the meta-analysis were impressive, finding that effective principals had a significant and positive impact on student test scores. Having an effective principal in the highest quarter of those studied compared with one in the lowest quarter should translate to the equivalent of an extra 2.9 months of math instruction and 2.7 months of English for students. That’s an amount of extra instruction that you’d expect if instruction were year-round.
“Across six rigorous studies estimating principals’ effects…” the report notes, “principals’ contributions to student achievement were nearly as large as the average effects of teachers identified in similar studies. Principals’ effects, however, are larger in scope because they are averaged over all students in a school, rather than a classroom.”
It was a formidable finding.
But after the report was released, a new analysis of measuring principal effects on student achievement, published in the Journal of Public Economics in 2024, demonstrated that there could be a number of different reasons for the findings reported by the six studies used in the meta-analysis in How Principals Affect Students and Schools. While they couldn’t demonstrate definitively that principals had not had the impact that was claimed, authors Brendan Bartanen, Aliza N. Husain, David D. Liebowitz put forward plausible alternative explanations for the results. “Value-added modeling requires researchers to make some assumptions about what they are measuring," says Grissom. “The authors of the 2024 article showed that using a different set of assumptions would lead to a different set of estimated effects for principals that aren’t as large as the ones we reported in How Principals Affect Students and Schools. And they make a pretty good argument that their assumptions were more plausible than the ones that the prior studies had relied on."
This is how science works—new tools, approaches, assumptions lead to new insights and questions. When the new analysis was published, Wallace discussed with Grissom, Egalité, and Lindsey whether we should issue a revised edition of the report. We collectively agreed that the methodology used in the meta-analysis of these six studies was executed correctly and the conclusions were accurate at the time. As Grissom explains, “The six studies we synthesized in 2021 acknowledged the potential limitations of extending this kind of modeling, which the field has largely accepted for measuring teacher effects, to the more indirect case of principals. They were all using the best methods of the time. It just turned out those methods may have been systematically misattributing to the principal some factors that aren’t under the principal’s control. We needed this new research by Bartanen, Husain, and Liebowitz to help everyone realize it.”
The newer analysis found that, rather than producing an average of almost three months of additional instruction in math and English, the effects of having a high performing principal instead of a low performing one would translate to about three weeks of additional mathematics instruction and a little more than one week of instruction in English. Although these magnitudes are smaller than in prior studies, the authors acknowledged that they can’t entirely rule out them being a bit larger. Regardless, Grissom points out that these more modest estimates still show that high quality principals are important. “If I was a state or district leader thinking about the levers I can pull to improve student learning,” he says, “I’d for sure sign up for an extra few weeks of learning multiplied across every kid in a school, year over year.”
The study by Bartanen and colleagues refines the use of an important measurement technique and highlights challenges not just for research but also for district and state policies that link hiring and promotion decisions to student test scores. We see this as important progress.
If I was a state or district leader thinking about the levers I can pull to improve student learning, I’d for sure sign up for an extra few weeks of learning multiplied across every kid in a school, year over year.
— Jason Grissom
The jury is now in that the results were overestimated, but the jury is still out on the extent to which they are overestimated. Why? This kind of quantitative analysis essentially involves “black box” measures of effects on student achievement. That is, you can describe the profile of the leader and name the outcomes for students, but there is limited clarity about what the leader actually has done that created those outcomes. Perhaps more importantly for the field, How Principals Affect Students and Schools synthesizes the overwhelming evidence, both quantitative and qualitative, of what principals do to positively influence outcomes for students, teachers, and families.
Seeking causality in school leadership research
How Principals Affect Students and Schools has a lot to say about how principals shape the conditions for teachers to support student learning. Specifically, the researchers found, effective principals are skilled in not only instructional leadership but also organizational development and people management. They exercise these skills through actively engaging teachers on instructional practice, building productive school climates, supporting professional collaboration and learning, and strategically managing personnel and resources. Much of this work takes place out of the public limelight. It requires investment in building systems in schools over time. It requires collecting and using data, maintaining strong relationships, giving good feedback, promoting communication and trust, maximizing use of time, and hiring and retaining well. Creating opportunities for students to learn is a complex and challenging business.
The technical challenge of capturing the outcomes of this complex work in a downstream, high stakes measure like test scores turns out to be a tough one. Bartanen, the lead author of the paper in the Journal of Public Economics, continues to work on cracking this nut. He notes that much of what a principal does cannot be reduced to a number. But, he says, principals are clearly an important input into what helps students achieve, along with teachers, family, community, and others. It’s like a puzzle or at any rate an empirical question: Can a principal’s causal input be quantified in the complicated equation articulating the combination of factors that lead to student achievement?
Principals are clearly an important input into what helps students achieve, along with teachers, family, community, and others.
— Brendan Bartanen
“It’s not like I think anybody’s sitting on the fence debating whether principals matter and waiting for a statistical breakthrough to decide yes or no,” he says. “But we're always making decisions about resources and investments with constraints. So getting at how much they matter could inform policymaking, resource allocation, and professional development. If we could conceptualize all of the variables and operators in the equation we might surface the system’s high leverage points.” Bartanen is committed to continuing work on the technical side of isolating the contributions of leaders. It’s a work in progress.
Where do we go from here?
At Wallace, we believe that there need to be many different approaches to measuring principals’ impacts. In our current Equity-Centered Pipeline Initiative (ECPI) we have funded a multi-institutional research team, anchored by the University of Wisconsin, to develop two tools that we think can help schools, districts, and states document and even measure the impacts of effective leaders in their schools. Outcomes include but expand beyond student achievement.
One tool is a school-level survey that unpacks and measures what principals actually do or effectuate in their schools—effectively opening up the black box regarding what leadership practices are occurring that may link to impact. The leader of the study, Rich Halverson from the University of Wisconsin, notes that “measurement for accountability is important. But we also want to measure things that provide principals with information they can use to improve their work and, ultimately, outcomes for students.” The tool measures the strength of a number of different areas of practice that principals are responsible for, such as building school culture and partnering with families and communities.
Measurement for accountability is important. But we also want to measure things that provide principals with information they can use to improve their work and, ultimately, outcomes for students.
— Rich Halverson
Another team is looking at the many ways that leadership practices are linked to the expansion of opportunities to learn (such as access to certified teachers and productive disciplinary practices) as well as achievement outcomes (such as test scores, graduation rates). Building off of a 2019 framework developed by the National Academies of Sciences, Engineering, and Medicine, the team, led by Alex Bowers at Teachers College, Columbia University, has catalogued about 4,000 relevant public datasets from ECPI districts and states, which they are integrating into a dashboard to measure progress on these outcome indicators. Bowers notes: “We’re envisioning a system that can demonstrate the range of impacts principals have, rooted in quantitative and empirical data. This will include student achievement. When principals dial up the effective leadership practices documented by this tool, we may be able to see how they ripple out, leading to improved learning opportunities and outcomes.”
Together, researchers at Wisconsin and Teacher’s College are hoping to make the connection between the strength of equity-centered leadership practices and the expansion of opportunity and achievement outcomes. We’ll be sharing their results when they are in. In the meantime, the technical quest for describing causal relationships of indirect influences continues. Stay tuned.
