Skip Navigation

Search Results: (16-23 of 23 records)

 Pub Number  Title  Date
NCEE 20090049 What to Do When Data Are Missing in Group Randomized Controlled Trials
This NCEE Technical Methods report examines how to address the problem of missing data in the analysis of data in Randomized Controlled Trials (RCTs) of educational interventions, with a particular focus on the common educational situation in which groups of students such as entire classrooms or schools are randomized. Missing outcome data are a problem for two reasons: (1) the loss of sample members can reduce the power to detect statistically significant differences, and (2) the introduction of non-random differences between the treatment and control groups can lead to bias in the estimate of the interventionfs effect. The report reviews a selection of methods available for addressing missing data, and then examines their relative performance using extensive simulations that varied a typical educational RCT on three dimensions: (1) the amount of missing data; (2) the level at which data are missing„Ÿat the level of whole schools (the assumed unit of randomization) or for students within schools; and, (3) the underlying missing data mechanism. The performance of the different methods is assessed in terms of bias in both the estimated impact and the associated standard error.
10/13/2009
NCEE 20090061 The Estimation of Average Treatment Effects for Clustered RCTs of Education Interventions
Reports in this series are designed for use by researchers, methodologists, and evaluation specialists to provide guidance in resolving or advancing challenges to evaluation methods. This paper examines the estimation of two-stage clustered RCT designs in education research using the Neyman causal inference framework that underlies experiments. The key distinction between the considered causal models is whether potential treatment and control group outcomes are considered to be fixed for the study population (the finite-population model) or randomly selected from a vaguely-defined universe (the super-population model). Appropriate estimators are derived and discussed for each model. Using data from five large-scale clustered RCTs in the education area, the empirical analysis estimates impacts and their standard errors using the considered estimators. For all studies, the estimators yield identical findings concerning statistical significance. However, standard errors sometimes differ, suggesting that policy conclusions from RCTs could be sensitive to the choice of estimator. Thus, a key recommendation is that analysts test the sensitivity of their impact findings using different estimation methods and cluster-level weighting schemes.
8/31/2009
NCEE 20094040 Technical Methods Report: Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs
In randomized control trials (RCTs) in the education field, the complier average causal effect (CACE) parameter is often of policy interest, because it pertains to intervention effects for students who receive a meaningful dose of treatment services. This report uses a causal inference and instrumental variables framework to examine the identification and estimation of the CACE parameter for two-level clustered RCTs. The report also provides simple asymptotic variance formulas for CACE impact estimators measured in nominal and standard deviation units. In the empirical work, data from ten large RCTs are used to compare significance findings using correct CACE variance estimators and commonly-used approximations that ignore the estimation error in service receipt rates and outcome standard deviations. Our key finding is that the variance corrections have very little effect on the standard errors of standardized CACE impact estimators. Across the examined outcomes, the correction terms typically raise the standard errors by less than 1 percent, and change p-values at the fourth or higher decimal place.
4/6/2009
NCEE 20094033 The Late Pretest Problem in Randomized Control Trials of Education Interventions
The Late Pretest Problem in Randomized Control Trials of Education Interventions, by Peter Schochet, addresses pretest-posttest experimental designs that are often used in randomized control trials (RCTs) in the education field to improve the precision of the estimated treatment effects. For logistic reasons, however, pretest data are often collected after random assignment, so that including them in the analysis could bias the posttest impact estimates. Thus, the issue of whether to collect and use late pretest data in RCTs involves a variance-bias tradeoff. This paper addresses this issue both theoretically and empirically for several commonly-used impact estimators using a loss function approach that is grounded in the causal inference literature. The key finding is that for RCTs of interventions that aim to improve student test scores, estimators that include late pretests will typically be preferred to estimators that exclude them or that instead include uncontaminated baseline test score data from other sources. This result holds as long as the growth in test score impacts do not grow very quickly early in the school year.
12/1/2008
NCEE 20084026 Technical Methods Report: Statistical Power for Regression Discontinuity Designs in Education Evaluations
Technical Methods Report: Statistical Power for Regression Discontinuity Designs in Education Evaluations examines theoretical and empirical issues related to the statistical power of impact estimates under clustered regression discontinuity (RD) designs. The theory is grounded in the causal inference and HLM modeling literature, and the empirical work focuses on commonly-used designs in education research to test intervention effects on student test scores. The main conclusion is that three to four times larger samples are typically required under RD than experimental clustered designs to produce impacts with the same level of statistical precision. Thus, the viability of using RD designs for new impact evaluations of educational interventions may be limited, and will depend on the point of treatment assignment, the availability of pretests, and key research questions.
8/11/2008
NCEE 20084018 Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations
Technical Methods Report: Guidelines for Multiple Testing in Impact Evaluations, presents guidelines for education researchers that address the multiple comparisons problem in impact evaluations in the education area. The problem occurs due to the large number of hypothesis tests that are typically conducted across outcomes and subgroups in evaluation studies, which can lead to spurious significant impact findings.
6/9/2008
NCES 1986223 High School and Beyond Postsecondary Education Transcript Study. Data File User's Manual, Contractor Report.
This document is intended as a data file user's manual for the dataset resulting from the High School and Beyond Postsecondary Education Transcript Study. The purpose of this study, conducted during 1984-85, was to provide reliable and objective information about the types and patterns of postsecondary courses taken by all members of the High School and Beyond (HS&B) 1980 senior cohort. Transcripts were requested from each school reported by sample members in their responses to the first and second follow-up surveys. Information from the transcripts, including terms of attendance, fields of study, specific courses taken, and grades and credits earned, were coded and processed into a system of data files designed to be merged with HS&B questionnaire data files. The Computer Assisted Data Entry System was used for processing the collected data into a four-level hierarchy consisting of data at the student, transcript, term, and course levels. The file includes specific data items in the student-level records including: physical tape position on the data record, a short description of the field contents, and the nature of the value stored in the field. Frequently distributions for all categorical variables on the student record are displayed in the enclosed codebook. The appendices, which make up over three-fourths of the document, contain: (1) a list of endorsing institutions; (2) postsecondary school codes in numerical and alphabetical order; (3) course subject codes in numerical order; (4) the data file record layout; (5) information on other HS&B data files available from the Center of Statistics; (6) sampling error and design effects; and (7) frequency distributions.
4/12/1988
NCES 1982218 A Classification of Secondary School Courses
Intended for use in collecting data on secondary school course offerings, this inventory of courses taught nationwide at the secondary level is organized in a classified arrangement. Each course is identified by a six-digit numerical code. The inventory of course titles was developed from three major sources--a sample of 52 public and private secondary school catalogs dated 1979 through 1981; statewide course lists from California, Washington, and Illinois; and recommendations from a panel of experts in secondary school curricula. The inventory includes instructions on its use and information about special features, a list of instructional program categories, cross references for closely related programs, and a title index. Courses are classified under 30 headings.
7/23/1982
<< Prev    16 - 23    
Page 2  of  2