Measuring school leaders' effectiveness: Final report from a multiyear pilot of Pennsylvania's Framework for Leadership

Title:	Measuring school leaders' effectiveness: Final report from a multiyear pilot of Pennsylvania's Framework for Leadership
Description:	This study examines the accuracy of performance ratings from the Framework for Leadership (FFL), Pennsylvania's tool for evaluating the leadership practices of principals and assistant principals. The study analyzed four key properties of the FFL: score variation, internal consistency, year-to-year stability, and concurrent validity. Score variation was characterized by the percentages of school leaders earning scores in different portions of the rating scale. To measure the internal consistency of the FFL, Cronbach's alpha was calculated for the full FFL and for each of its four categories of leadership practices. Analyses of score stability used data on FFL scores of school years across two years to calculate Pearson’s correlation coefficient. Concurrent validity was assessed through a regression model for the relationship between school leaders' estimated contributions to student achievement growth and their FFL scores. This report is based primarily on the 2013/14 pilot in which 517 principals and 123 assistant principals were rated by their supervisors; an interim report examined data from the 2012/13 pilot year. The study finds that the FFL is a reliable measure, with good internal consistency and a moderate level of year-to-year stability in scores. The study also finds evidence of the FFL's concurrent validity: principals with higher scores on the FFL, on average, make larger estimated contributions to student achievement growth. Higher total FFL scores and scores in two of the four FFL domains are significantly or marginally significantly associated with both value-added in all subjects combined and value-added in math specifically. This evidence of the validity of the FFL sets it apart from other principal evaluation tools: No other measures of principals' professional practice have been shown to be related to principals' effects on student achievement. However, in both pilot years, variation in scores was limited, with most school leaders scoring in the upper third of the rating scale. As the FFL is implemented statewide, continued examination of evidence on its statistical properties, especially the variation in scores, is important.
Online Availability:	Download, view and print the report as a pdf file. (933KB) Download, view and print the “Frequently Asked Questions” document about the study (484KB)
Cover Date:	January 2016
Web Release:	January 21, 2016
Print Release:	January 21, 2016
Publication #:	REL 2016106 General Ordering Information
Center/Program:	REL
Associated Centers:	NCEE
Authors:	Moira McCullough, Stephen Lipscomb, Hanley Chiang, Brian Gill, and Irina Cheban: Mathematica Policy Research
Type of Product:	Making Connections
Keywords:	Control of Schools Principals Schools • management of States • principals, characteristics of
Questions:	For questions about the content of this Making Connections, please contact: Amy Johnson.