A.3 Study Design
A.3.1 Sampling
The ELS:2002 base-year sample design began with a nationally representative, two-stage stratified probability sample. The first stage of selection was schools; schools were selected with probability proportional to size (PPS). The public school sample was stratified by the nine U.S. Census divisions and by urbanicity (metropolitan status of urban, suburban, or rural). Private schools (Catholic and other private) were stratified by four levels of geography (Census region) and urbanicity; private schools were oversampled. The target sample size was 800 schools. Cooperation was sought from 1,220 eligible selections. The realized sample comprised 750 participating 10th-grade schools (67 percent participation rate). The second stage of selection was students. Of 17,590 sampled students in the schools, 15,360 students participated. Some groups (e.g., Asians, students in nonpublic schools) were oversampled. The weighted student response rate was 87 percent.
The first follow-up returned to the same schools to seek their cooperation, and to base-year sophomore respondents and a sample of base-year nonrespondents, regardless of whether they had remained in the base-year school. Although 5 of the 750 base-year schools were ineligible because they no longer enrolled ELS:2002 sample members or seniors, of the eligible schools, 700 (93 percent) participated. Overall, there were 16,520 sample members (students, dropouts, homeschooled, or early graduates), of whom 14,990 participated. This analysis uses the sample of 14,710 students who were sophomores in 2001–02, participated in both the BY and F1 interviews, completed the mathematics assessment in the BY and F1 interviews, and had complete transcript information for the 2002–03 and 2003–04 academic years. Students were in the same school in the base year and follow-up survey.
A.3.2 Weighting and Imputation
- Weighting.
A number of weights are included on the ELS:2002 data file to compensate for unequal probabilities of selection of schools and students into the base-year sample and to adjust for the fact that not all schools and students selected into the sample actually participated. The analyses in this report are weighted with the panel weight (F1PNLWT), which accommodates analyses using sample members who participated in both the base year and first follow-up.
- Imputation.
For key classification variables used in this analysis, missing data were replaced with imputed values. These include: sex, race/ethnicity, family composition, educational expectations, and socioeconomic status. Single imputation (by means of a weighted sequential hot deck procedure) was implemented for missing key questionnaire variables. Multiple imputation of the mathematics ability estimate theta (theta is the point on the test scale that marks the ability of the test taker) was used to treat missing assessment data. Although (for several classes of respondents) missing test scores were imputed in ELS:2002, imputed test data have not been used in this report. Only students with two unimputed test scores are included in the analysis sample.
A.3.3 Base-Year and First Follow-up Response Rates
- Base-year Response Rates.
Of 1,220 eligible contacted schools, 750 participated in the study, for an overall weighted school participation rate of approximately 68 percent (62 percent unweighted). Of 17,590 selected eligible students, 15,360 participated, for a weighted student response rate of approximately 87 percent.2 (School and student weighted response rates reflect use of the base weight [design weight] and do not include nonresponse adjustments.) School and student unit nonresponse bias analyses were performed, as well as an item nonresponse bias analysis for the questionnaires. The school-level bias due to nonresponse prior to and after computing weights was estimated based on the data collected from both respondents and nonrespondents, as well as sampling frame data. At the unit level (but not the item level), weighting techniques were employed to reduce detected bias; after final nonresponse adjustments, the remaining relative bias ranged from 0 percent to 0.2 percent for schools and from 0 percent to 0.07 percent for students. For details of the bias analyses, see the Education Longitudinal Study of 2002: Base Year Data File User's Manual (Ingels et al. 2004). Unweighted and weighted school-level response by stratum is summarized in table A-1. Table A-2 summarizes base-year response rates by instrument.
- First Follow-up Response Rates.
First follow-up weighted response rates are reported at the student level only (the school sample was not strictly representative of the nation's high schools with 12th grades in 2003–04). Overall, 14,990 of 16,520 sample members participated, for a weighted response rate of 89 percent. Further details of first follow-up coverage and completion rates are provided in table A-3.
- High School Transcript Response and Coverage Rates.
A total of 1,550 out of 1,950 schools (base-year schools and transfer schools) participated in the request for transcripts for an unweighted participation rate of 79 percent. The base-year school weighted response rate is 95 percent. The course offerings response rate for base-year schools is 88 percent. Ninety-one percent of the entire student sample have some transcript information (14,920 out of 16,370). Note that for transcripts, a coverage rate-indicating the number of students who participated in one of the two rounds who have transcript data-is given rather than a response rate. Table A-4 provides coverage rates for base-year students in the high school transcript study.
A.3.4 Quality of Estimates: Reliability and Validity of ELS:2002 Questionnaire and Transcript Data
Most of the items used in the ELS:2002 questionnaires were taken from prior studies, particularly HS&B and NELS:88. Given their past use with large, nationally representative samples, their measurement characteristics are well established. A number of data quality studies have been conducted using these items. Interested readers should see, in particular, Fetters, Stowe, and Owings (1984), Kaufman and Rasinski (1991), and McLaughlin and Cohen (1997). Data quality analyses for the subset of new questionnaire items used in ELS:2002 (as well as the reading and mathematics assessments) can be found in the base-year field test report (Burns et al. 2003). The base-year and base-year to first follow-up data documentation manuals (Ingels et al. 2004, 2005) also address issues of questionnaire and assessment data quality for both the ELS:2002 baseline and its first follow-up, while Bozick et al. (2006) address similar issues for the high school transcript component of the study. Data quality for the mathematics assessments is discussed in section A.5 of this appendix. While transcript data are assumed to be superior to student self-report data (the degree of difference between records sources and questionnaire responses is set out in Fetters, Stowe and Owings [1984]), archival records are not infallible data sources. Apart from problems of nonresponse (although response rates for the ELS:2002 transcript collection were high), a major records-gathering problem with a mobile longitudinal cohort is incompleteness of data. Some 14 percent of transcript respondents do not have 4 "complete" years of high school records information (Bozick et al. 2006). However, the problem of incompleteness in part reflects the fact that some records are necessarily incomplete (dropouts and those who failed to graduate with their cohort members by definition have incomplete high school records). Apart from the necessarily incomplete records of students who dropped out or did not advance in modal sequence of the cohort, full records were substantially more difficult to obtain for transfer students than for students who did not move to a new school. Since dropouts, held back, and transfer students are not represented in the analyses in this report, these nonresponse factors are substantially mitigated.
A.3.5 Survey Standard Errors
Because the ELS:2002 sample design involved stratification, the disproportionate sampling of certain strata, and clustered (i.e., multistage) probability sampling, the resulting statistics are more variable than they would have been if they had been based on data from a simple random sample of the same size. In all analyses in this report, standard errors were adjusted for the clustered and stratified sampling design using Taylor-series linearization methods.
2 Stage 1 (school) response rates can be multiplied by stage 2 (student) response rates for a combined two-stage response rate: 68 percent * 87 percent = 59 percent.