Title: | Stated Briefly: How methodology decisions affect the variability of schools identified as beating the odds |
Description: | This "Stated Briefly" report is a companion piece that summarizes the results of another report of the same name. Schools that show better academic performance than would be expected given characteristics of the school and student populations are often described as "beating the odds" (BTO). State and local education agencies often attempt to identify such schools as a means of identifying strategies or practices that might be contributing to the schools' relative success. Key decisions on how to identify BTO schools may affect whether schools make the BTO list and thereby the identification of practices used to beat the odds. The purpose of this study was to examine how a list of BTO schools might change depending on the methodological choices and selection of indicators used in the BTO identification process. This study considered whether choices of methodologies and type of indicators affect the schools that are identified as BTO. The three indicators were (1) type of performance measure used to compare schools, (2) the types of school characteristics used as controls in selecting BTO schools, and (3) the school sample configuration used to pool schools across grade levels. The study applied statistical models involving the different methodologies and indicators and documented how the lists schools identified as BTO changed based on the models. Public school and student data from one midwest state from 2007-08 through 2010-11 academic years were used to generate BTO school lists. By performing pairwise comparisons among BTO school lists and computing agreement rates among models, the project team was able to gauge the variation in BTO identification results. Results indicate that even when similar specifications were applied across statistical methods, different sets of BTO schools were identified. In addition, for each statistical method used, the lists of BTO schools identified varied with the choice of indicators. Fewer than half of the schools were identified as BTO in more than one year. The results demonstrate that different technical decisions can lead to different identification results. |
Online Availability: | |
Cover Date: | April 2016 |
Web Release: | April 6, 2016 |
Print Release: | April 5, 2016 |
Publication #: | REL 2016119 General Ordering Information |
Center/Program: | REL |
Associated Centers: | NCEE |
Authors: | Coby Meyers and Amy Proger: American Institutes for Research; Yasuyo Abe, Phyllis Weinstock, and Vincent Chan: IMPAQ International. In collaboration with the Midwest Beating the Odds Research Alliance |
Type of Product: | Stated Briefly |
Keywords: | |
Questions: |
For questions about the content of this Stated Briefly, please contact: Amy Johnson. |