Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation

Title:	Comparing Impact Findings from Design-Based and Model-Based Methods: An Empirical Investigation
Description:	This report compares empirical results from different approaches to analyzing data from randomized controlled trials (RCTs). It focuses on how impact estimates compare between recently-developed design-based methods and traditional model-based methods. Design-based methods use the potential outcomes framework and known features of study designs to connect statistical methods to the building blocks of causal inference. They differ from model-based methods that have commonly been used in education research, including hierarchical linear model (HLM) methods and robust cluster standard error (RCSE) methods for clustered designs. This study re-analyzes nine past RCTs in the education area using both design- and model-based methods. The study finds that model-based and design-based methods yield very similar impact estimates and levels of statistical significance, especially when the underlying analytic assumptions (e.g., weights used to aggregate clusters and blocks) are aligned.
Online Availability:	Download, view and print the report as a pdf file. (3MB)
Cover Date:	July 2017
Web Release:	July 25, 2017
Print Release:	July 25, 2017
Publication #:	NCEE 20174026 General Ordering Information
Center/Program:	NCEE
Authors:	Tim Kautz, Peter Z. Schochet, and Charles Tilley: Mathematica Policy Research
Type of Product:	Research Report
Keywords:	Randomized Control Trials
Questions:	For questions about the content of this Research Report, please contact: Amy Johnson.