Methodological Research
Below is a collection of papers and presentations that summarize methodological and topical research produced by NCES. Noted research below helped to inform the development and direction of High School and Beyond surveys, instruments, and measures collected.
Bolded authors indicate NCES staff.
ABSTRACT: For its recent field test for the First Follow-up of the High School and Beyond Longitudinal Study of 2022 (HS&B:22), NCES experimented with different data collection strategies for parent surveys. In the Base Year (2022–23 school year), parent response rates were the lowest among the 5 different respondent groups for the study. That is, they were lower than those of high school students, their math teachers, school counselors and school principals. Because parents provide important demographic data about their high schooler sample members, eliminating the parent questionnaire entirely would leave data gaps in key variables. A frequent reason for survey refusal reported to interviewers is that parents are "too busy" or "don't have the time" to participate. While an abbreviated questionnaire is often offered toward the end of the data collection window for NCES sample surveys as a refusal conversion technique, few studies have tried drastically shortening the questionnaire at the outset of data collection. Considering this, NCES explored options for dramatically minimizing the time burden requested to test whether reducing the burden might encourage more parents to respond.
To test procedures to boost response rates and/or minimize missingness for key variables, NCES experimented with data collection utilizing micro-questionnaires in paper-and-pencil interview (PAPI), computer-assisted telephone interview (CATI) and web-based interview modes. Results are discussed in terms of response rates, data quality and potential for bias. Implications for further implementation of micro-surveying sum up the discussion.
ABSTRACT: The High School Longitudinal Study of 2009 (HSLS:09) is a nationally representative study of 2009 ninth-graders' important transitions through high school and into young adult outcomes. Information about motivations, expectations, and attitudes is added to rich demographic data in order to inform research about educational pathways. HSLS:09 is the fifth in a series of secondary longitudinal studies conducted by the National Center for Education Statistics (NCES) since the early 1970s.
The current paper will present findings using newly released NCES data from the HSLS:09's Postsecondary Education Administrative Records (PEAR) collection (to be released in fall 2023). This dataset has been compiled by administrative matching with the US Department of Education's National Student Loan Data System (NSLDS) and with the National Student Clearinghouse (NSC) to understand postsecondary enrollment and degree completion through July 2021, which was eight years after most students in the HSLS:09 cohort left high school. These data provide an updated, more comprehensive picture of postsecondary degree attainment among the 2009 ninth-grader population.
ABSTRACT: Over several decades, the National Center for Education Statistics has collected data from high school students and their families, including demographic information such as gender. In previous iterations of longitudinal surveys, sample members may have updated their demographics between rounds. It is not known the extent to which these updates reflect measurement error or changes in self-identification of sample members. In the case of gender, NCES evaluated and employed a two-step measure of sex at birth and gender identity on a later round of a longitudinal survey, the High School Longitudinal Study of 2009 (HSLS:09). To understand the potential for measurement error, several types of data were compared, including sample member responses across modes, between rounds, and administrative records. Analyses in this paper focus on whether measurement error decreased with the two-step method, which is a primary goal of collecting gender information in this way.
ABSTRACT: In the High School Longitudinal Study of 2009, conducted by the National Center for Education Statistics, measures of sexual orientation and gender identity (SOGI) were added in the Second Follow-up, conducted in 2016. The panel of 2009 high school freshmen had already participated in three rounds of data collection in 2009, 2012, and 2013. These potentially sensitive SOGI measures were extensively examined via cognitive testing and field testing prior to their addition to national survey instruments. After national data collection was completed, paradata and metadata were used to examine whether these items led to data quality concerns. This presentation will share results of these analyses, including investigations of breakoffs, item-level nonresponse, and time spent on item screens. Results will be compared to those of extant sensitive items in the survey in order to make conclusions about item functioning.
ABSTRACT: Over the past few years, NCES has added or been in the process of adding survey items on gender identity to several of its national surveys, including the High School Longitudinal Study of 2009 (HSLS:09 ) and Baccalaureate and Beyond (B&B ) surveys. As part of this process of item development, cognitive testing has been conducted across these study populations in order to understand the performance of the items. The discussion here will center on how the cognitive testing was conducted, the influence of HSLS:09 testing and findings on B&B initial items and testing processes, and lessons that NCES has learned about item development for gender minority populations.
ABSTRACT: Dr. Christopher overviews efforts made through the U.S. Department of Education's National Center for Education Statistics (NCES) to include sexual orientation and gender identity demographic items in their sample surveys of adults. In particular, she overviews the National Postsecondary Student Aid Study (NPSAS), the Beginning Postsecondary Students Longitudinal Study (BPS), the Baccalaureate and Beyond Longitudinal Study (B&B), and the High School Longitudinal Study of 2009 (HSLS:09) which together have collected sexual orientation and gender identity demographic data from over 100,000 postsecondary students, postsecondary degree recipients and young adults. These SOGI data are important sources for research to understand high school, postsecondary and labor market experiences for LGBTQ+ individuals.
ABSTRACT: The National Center for Education Statistics (NCES) has been working in recent years to improve declining response rates in its sample surveys. Across multiple longitudinal and cross-sectional surveys, several methods of adaptive design have been tested. Using models to determine likelihood to introduce bias should sample members not respond, NCES has effectively discerned which sample members to target for special intervention, in order to boost response rates. However, it is clear that sample members' propensity to respond is not always strongly correlated with their likelihood to introduce bias if they do not respond, necessitating more research on tailored ways to encourage response. Incentives and other treatments thus become an important area of investigation. This presentation describes the adaptive design approach that NCES used in the High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up Field Test, conducted in spring and summer, 2015. This field test examined the effectiveness of several types of incentive treatments to encourage sample member response. The comparative effectiveness of four treatment methods will be discussed: 1) prepaid incentives, 2) response-contingent incentives, 3) incentive boosts, and 4) an abbreviated interview. The paper will describe the results from the adaptive design experiment and the plans for implementation of the findings in full-scale data collection in 2016.
ABSTRACT: This presentation highlights some of the adaptive design strategies used in recent NCES longitudinal surveys. We review model selection, treatments, and results from the 2012 follow-up of the Education Longitudinal Study of 2002 (ELS:2002), the 2008-12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12), and the 2012–14 Beginning Postsecondary Students Longitudinal Study (BPS:12/14). We discuss the potential for using adaptive design to reduce survey nonresponse bias and note considerations for implementation.
ABSTRACT: Elise Christopher and Ted Socha of the National Center for Education Statistics will be discussing how responsive design (RD) models have been implemented in two of the Center's nationally-representative longitudinal survey data collections: the Education Longitudinal Study of 2002 (ELS:2002) and the 2008–12 Baccalaureate and Beyond Longitudinal Study (B&B:08/12). Topics will include the following: (1) choice of RD models, (2) how each RD model was applied, (3) implications of using RD for monetary and non-monetary incentives, (4) substantive and paradata model variables, (5) dealing with missingness in substantive variables, and (6) what their preliminary results mean for nonresponse bias reduction.