Technical Report:

Sample Exclusion in NELS:88: Characteristics of Base Year Ineligible Students; Changes in Eligibility Status After Four Years

May 1996

Introduction

Overview of NELS:88. The information reported here is taken from the National Education Longitudinal Study of 1988 (NELS:88). The base year of NELS:88 was the first stage of a major longitudinal effort designed to provide trend data about critical transitions experienced by students as they leave elementary school and progress through high school and into postsecondary institutions or the work force. The base year study, conducted in the spring term of the 1987-88 school year, selected 26,432 potential eighth grade participants, of whom 24,599 were successfully surveyed in 1,052 public and private schools. Additional data were gathered from eighth graders’ parents, teachers, and principals.

The first follow-up in 1990 provided the first opportunity for longitudinal measurement of the 1988 baseline sample. It also--after sample freshening/1--provided a comparison point to high school sophomores ten years before, as studied in High School and Beyond (HS&B). One of the chief goals of the NELS: 88 design has been to capture in a longitudinal data set the entire subpopulation of school dropouts from within a high school entry cohort.

The second follow-up took place in 1992, when most sample members entered the second term of their senior year. The second follow-up provides a culminating measurement of learning in the course of secondary school, and also collects information that will facilitate investigation of the transition into the labor force and postsecondary education after high school. Freshening the NELS:88 sample to represent the twelfth grade class of 1992 makes trend comparisons with the senior cohorts that were studied in the National Longitudinal Study of the High School Class of 1972 (NLS-72) and HS&B possible. The NELS:88 second follow-up resurveyed students who were identified as dropouts in 1990, and identified and surveyed those additional students who lefl school after the first follow-up.

In all three in-school rounds, students were asked to complete a student questionnaire, taking up to one hour to do so, and an 85-minute achievement battery comprising tests in reading, mathematics, social studies and science. Except for some experimental constructed response items in 1992 (on which, see Pollack and Rock, 1996), the cognitive tests were in a multiple choice format.

The NELS: 88 third follow-up took place in the spring of 1994, with most sample members in postsecondary education or in the labor force. Base year ineligible students who remained ineligible in 1990 and 1992 were not included in the 1994 round. Base year ineligible students who had become eligible in 1990 or in 1992 are represented in the 1994 sample. A major goal of the 1994 round is to provide data for trend comparisons with NLS-72 and HS&B, and to continue cross-wave comparisons with previous NELS:88 rounds. The 1994 survey provides a basis for assessing how many dropouts have returned to school and by what route, and measures the access of dropouts to vocational training programs and to other postsecondary institutions. A fourth follow-up is scheduled for 1998 or 2000.

Sample Undercoverave in NELS:88. Errors in surveys are generally classified into two types: sampling and nonsampling errors. Sampling errors consist of differences between the sample and the population that are attributable to chance. Nonsampling errors in surveys and censuses may be grouped in three broad categories: (1) measurement or response errors, (2) errors due to nonresponse or missing data, and (3) coverage errors. This report is concerned with coverage error. Simply stated, coverage error is the failure to include the entire universe of interest in the population to which the sample data are projected. Coverage error is normally caused by an incomplete or out-of-date sampling frame (for example, school lists or student rosters that are inaccurate or have not been updated), but it may also arise from eligibility rules that are inappropriate or not reliably and validly applied. In addition, the decision to exclude a portion of the universe as not of interest may always be queried and must always be justified. The issue of coverage ultimately leads back to the question of how the universe of interest should be defined.

There are many motivations for excluding categories of students by design from assessments and surveys. Sometimes students are excluded because of the practical and cost difficulties of including them, even though, conceptually, they are part of the population of interest. In other instances exclusion is based on the belief that the assessment (or questionnaire) would not be valid for a student within some given classification, either because the student’s disability or limited proficiency in English would not permit accurate measurement using the assessment instruments, or because the student’s classification is tied to a separate curriculum with distinct curricular goals such that the content of the assessment would be inappropriate. If a student receives a different curriculum from the one an achievement test measures, then the argument for regarding that student as legitimately not part of the defined “population of interest” would seem to be compelling. Indeed, some students with disabilities -- the severely impaired --- are in a separate curriculum. Most are not, nor, by and large, are LEP students although the unique features of bilingual-bicultural education programs should be taken into account. There is also an issue of who interprets exclusion criteria once they are set. Since assessments and surveys depend on school personnel to apply eligibility rules, the school-level incentives and disincentives to include or exclude particular categories of students must also be considered, as well as the schemes of classification and culture of the school that will influence the judgments of those implementing exclusion guidelines.

This methodology monograph addresses issues of student sample exclusion in NELS:88, particularly the problem posed by eighth graders who were declared ineligible for the study owing to factors -- physical or mental disabilities, or lack of proficiency in the English language -- deemed by their schools to pose a significant barrier to participation. In the base year, 5.4 percent of eighth graders were so excluded. (The weighted proportion of the sample that was excluded is 4.7 percent.) A sample of the excluded students was followed in subsequent rounds, and eligibility status reassessed. A substantial number of excluded 1988 eighth graders recentered the NELS: 88 sampling frame as eligible in 1990 and 1992. Organization of This Report. This report is divided into five chapters, a bibliography, and five appendices:

Chapter 1:

provides background on base year student sampling;

discusses the exclusion of students as a potential source of undercoverage;

discusses the reliability and validity of exclusion criteria;

describes the objectives and the implementation of the first follow-up Base Year Ineligibles study, and the second follow-up Followback Study of Excluded Students;

Chapter 2:

describes the characteristics of the base year ineligibles, specifically, the proportions belonging to various racial/ethnic or gender groups;

describes the proportion having selected school outcomes such as persistence in school, remaining in or falling behind modal grade progression;

Chapter 3:

depicts changes in ineligibility status between 1988 and 1992;

Chapter 4:

discusses other possible sources of undercoverage in the NELS:88 sample;

Chapter 5:

outlines conclusions drawn from this study and recommendations the Bibliography: for future studies;

details references used in preparing the report;

Appendix A:

provides supplemental tables that include data for all figures and tables reported in chapters 1-5 and additional bivariate statistics not reported in the main text;

Appendix B:

contains examples of screeners used in the redetermination of eligibility;

Appendix C:

provides information about eligibility criteria employed in HS&B and NAEP;

Appendix D:

provides technical notes on the sample, approach to analysis and survey methodology;

Appendix E:

supplies a glossary of technical terms used in this report.

FOOTNOTE:

[1] The process of “freshening” added spring term 1990 sophomores who were not in the base year sampling frame, either because they were not in the country or because they were not in eighth grade in the spring term of 1988. The 1990 freshening process that provided a representative NELS:88 sophomore cohort was repeated in 1992 in order to provide a NELS:88 senior cohort, that is, a nationally representative sample of students enrolled in twelfth grade in the spring term of 1992.

Download/view the full report in a PDF file.(3,084K)

Help with PDF files

For more information about the content of this report, contact Peggy Quinn