High School Longitudinal Study of 2009 (HSLS:09)

5. Data Quality and Comparability

Sampling Error

Analyses with HSLS:09 data should use statistical software that can calculate (a) BRR variance estimations using replicate weights and an associated analytic weight; or (b) linearization variance estimations through a Taylor series approximation using only the analytic weight. BRR weights are constructed to capture the variance associated with the sampling information and, along with appropriate software, provide an alternative to the linearization method. Linearization variance estimation requires software that constructs a first-order Taylor series approximation of the statistic being analyzed (e.g., the mean) and data sources containing the analytic stratum and primary sampling unit (PSU) identifiers as well as a single analytic weight (see, e.g., Binder 1983; Woodruff 1971). In contrast, BRR variance estimation does not require knowledge of the analytic strata and PSUs and instead only requires a large set of replicate weights and the main analytic weight.

Design effects. A total of 89 estimates from HSLS:09 were used in the design effects analysis: 22 school–level variables from the administrator and counselor questionnaires; 37 items from the student questionnaire plus one mathematics achievement score (theta); and 29 parent questionnaire items. The items were chosen using six criteria: (1) representation from the school–level instruments (administrator and counselor) and the student–level instruments (student and parent); (2) HSLS:09 variables common to the ELS:2002 base–year design effects analysis; (3) variables identified for the First Look report; (4) substantively important variables to NCES; (5) variables included in several other NCES studies, such as ELS:2002, NELS:88, and the National Postsecondary Student Aid Study (NPSAS); and (6) randomly selected items to ensure coverage of all sections of the instruments.

Nonsampling Error

Nonresponse Error. Both unit nonresponse (nonparticipation in the survey by a sample member) and item nonresponse (missing values for a given questionnaire/test item) have been evaluated in the base–year survey of HSLS:09.

Base–year unit nonresponse. HSLS:09 schools were classified as respondents if the school administrator permitted student data collection. The overall weighted school response rate for HSLS:09 was 56 percent. For sampled students, the weighted response rate exceeded the 85 percent threshold (86 percent); nevertheless, certain domains (e.g., school type, region, student sex, student race/ethnicity) were flagged for bias analysis.

Base year school–level nonresponse bias analysis. The purpose of this analysis was to determine the extent to which sampled units differed from nonsampled ones. Information was obtained from 66 percent of nonparticipating schools using an abbreviated questionnaire during an interview with the school administrator or contact at the district/diocese level. The abbreviated questionnaire, in combination with the NCES sampling frame items, netted a total of 15 variables for the school nonresponse bias analysis, including school type, region of the country, metropolitan designation, size of the school, ninth- grade enrollment count, and number of full-time teachers. Prior to adjusting the weights for nonresponse bias, 46 percent of the tests showed significant levels of bias, with a median absolute relative bias of 12 percent. Following adjustment of the weights for nonresponse, only 20 percent of the tests showed significant levels of bias, and the median absolute relative bias was reduced to 6 percent.

Base year student-level nonresponse bias analysis. The overall weighted response rate exceeded 85 percent for the HSLS:09 student sample (86 percent); however, the weighted response rates for certain domains fell below the threshold level; thus a nonresponse bias analysis was required. For the analysis, some information for nonresponding students, such as race/ethnicity and sex, was available from the school enrollment lists. School characteristics were also used in the analysis. In total, 17 variables were used. Approximately 18 percent of the 60 statistical tests identified significant bias, with the median absolute relative bias equal to 1 percent before adjustments were made. Following adjustments for nonresponse, no tests showed significant bias and the median absolute relative bias was reduced to zero.

Base year student-level contextual nonresponse bias analysis. The weighted response rates for the providers of student contextual information (science teacher, math teacher, and parent) all fell below 85 percent. Science and math teachers had response rates of 70 and 72 percent, respectively, while parents had a 76 percent response rate. Nevertheless, information on nonresponding teachers and parents was not available for either weight adjustment or for the nonresponse bias analysis. Student and school characteristics were thus used in the student-level nonresponse bias analysis for the contextual analyses. In total, 17 variables were used in the student nonresponse bias analysis. Bias was detected in 33 percent of the 60 tests implemented with the weight linking the student and science teacher and in 23 percent of the 60 tests implemented with the weights linking the student with the mathematics teacher or parent, respectively. Adjusting the weights for nonresponse reduced the observed bias, although some bias was still observed for all three contextual weights.

First follow–up nonresponse bias of student data. In keeping with the NCES statistical standards, nonresponse bias analyses were performed for first follow–up student responses at the student level, because the overall weighted response rate was 82.0 percent. Students who completed a substantial portion of the questionnaire were classified as a respondent, regardless of their level of participation in the mathematics assessment. In total, 17 variables were used for the student nonresponse bias analysis. Approximately 31.8 percent of the 66 statistical tests identified bias significantly greater than zero at the 0.05 significance level prior to adjusting the weights for nonresponse. After adjustment, no levels of bias were detectable at the 0.05 level of significance and the median absolute relative bias was reduced by 72.3 percent. Nonresponse bias was also evaluated in student items available for a longitudinal analysis. The overall weighted response rate for students with responses in the first follow–up and the base year was 74.3 percent. A total of 17 variables were used for the student longitudinal nonresponse bias analysis. These 17 variables resulted in 66 comparisons (tests). Bias was detected for 33.3 percent of the 66 tests implemented with the student longitudinal weight. After applying the nonresponse adjustments, no bias was statistically significant in any of the 66 tests. A 79.0 percentage point reduction was also seen in the median absolute relative bias.

First follow-up nonresponse bias of parent data. The overall parental weighted response rate for the students randomly selected for the first follow-up parent subsample was 72.5 percent. Information on the nonresponding parents, however, was not available for either weight adjustment or for the nonresponse bias analysis. Consequently, student and school characteristics used in the student-level nonresponse bias analysis were used for the student home–life contextual analyses. In total, 17 variables were used for the student-level contextual nonresponse bias analysis, including characteristics known for the base-year schools where the students were first selected for the study. Bias was initially detected for 25.8 percent of the 66 tests implemented with the first follow-up student home-life contextual weight.

After adjusting the weights, no tests were found to identify significant levels of bias. Also, the median relative bias was reduced by 89.7 percentage points. The weighted response rate for the student first follow-up home-life contextual subsample was 72.5 percent. Accounting for students and parents in the subsample who responded to the base year and first follow-up, the weighted response rate was reduced by 8.3 percentage points or 64.2 percent. The student home-life contextual nonresponse bias analysis initially identified 37.9 percent of the 66 tests as having significant levels of bias at the 0.05 level using the contextual longitudinal weight. After adjusting the weights, only 3 percent of the statistical tests produced significant results. Additionally, the median relative bias was reduced by 90.4 percentage points. A responsive design was implemented in the HSLS:09 first follow-up as one additional method for reducing nonresponse bias in the contextual information for students in the parent subsample. Through the use of propensity models, the parent cases with low likelihood of response (i.e., low propensity) were identified and targeted for additional recruitment efforts. Overall, approximately 42.4 percent of the categories initially showed an estimated bias that was statistically significant. Consequently, after the inclusion of low-propensity cases, 25.8 percent of the categories show estimated bias to be statistically significant.

2013 update and high school transcript nonresponse. In calculating response rates for the 2013 Update, there were two types to consider. The unconditional response rate is the response rate calculated with no exceptions for the temporarily out of scope, unfielded cases; only the deceased are removed from the study denominator. The unconditional response rate supports statistical description and estimation. The conditional response rate removes all four categories of temporary out of scope students from the denominator. The conditional response rate is a measure of the methodological success of the study, of what its data collection effort was able to accomplish. The 2013 Update data collection ended with a 73.1 percent weighted response rate. For transcripts, an 87.7 percent weighted response rate was achieved.

Second follow-up survey nonresponse. The weighted response rate for the second follow-up fell below 85 percent. Therefore, in accordance with NCES statistical standards, the data were subjected to bias analysis. Unit nonresponse bias analyses were conducted for each set of respondents that corresponded to one of the seven analytic weights: the five second follow-up weights and the two supplemental teacher weights for the 2013 Update. The following 15 categorical variables were used to assess unit nonresponse bias: school type, charter school status, ninth-grade enrollment by race, total school enrollment, ninth-grade enrollment, number of full-time teachers, student-to-teacher ratio, census region, school urbanicity, school grade range, religious affiliation of school, secondary status of school, state of school, sex, and race. (Note that several of the 15 variables are derived from sampling frame data and are therefore not available in either restricted-use or public-use files.) These 15 variables in total comprise 67 categories. For each category, estimates of bias were calculated and statistical significance tests conducted. Further information on the results of this nonresponse bias analysis may be found in the publication HSLS:09 Base-Year to Second Follow-Up Data File Documentation.

Item nonresponse. Item response rates measure the proportion of responses obtained for a particular question from respondents who were supposed to answer the question. Item response rates differ from a unit response rate, which measures the proportion of eligible sample members among those selected for the study who actually participate. As with unit nonresponse bias, item nonresponse bias occurs when items that should have a valid response are left blank, which affects the results produced from the data. A weighted item response rate among study participants less than 85 percent, calculated with the final analytic weight as in the HSLS:09 base year, was used to identify first follow-up variables for the item nonresponse bias analysis.

Item nonresponse bias analysis. Item nonresponse bias was evaluated for the questions having low levels of item response (less than 85 percent). The proportion of items requiring a nonresponse bias analysis varied across survey components. In the school questionnaire, 16 percent of items had response rates below 85 percent (79 out of 481 items). In the student questionnaire, 3 percent of items required nonresponse bias analyses (10 out of 376 items), as did 9 percent (16 out of 178 items) in the science teacher questionnaire; 14 percent (21 out of 152 items) in the math teacher questionnaire; and 26 percent (70 out of 266 items) in the parent questionnaire. The higher proportion of items requiring a nonresponse bias analysis in the parent questionnaire is partially accounted for by the abbreviated questionnaire used during nonresponse conversion. All study items with a weighted response rate less than 85 percent among the study participants were classified as having high item-nonresponse and were included in the item nonresponse bias analyses for first follow–up. Almost 78 percent of the item-nonresponse bias analysis variables (28 of 36 items) had a weighted item response rate of at least 80 percent, and over 54 percent of the item–nonresponse bias analysis variables (18 of 33 items) had a weighted item response rate of at least 60 percent.

Top

Item Imputation. In the base–year survey, HSLS:09 variables in general did not suffer from high levels of item nonresponse. Nevertheless, key analytic variables were identified for item imputation to facilitate complete–case analysis on data obtained from the participating ninth–grade students. Single–value imputation was used to replace missing responses for 18 key analytic variables from the student and parent questionnaires. These variables included important demographic variables (e.g., student’s race/ethnicity); student and parent educational expectations; parent’s relationship to the ninth–grader, highest level of education, employment status, and recent occupation; and family income. Additional variables were considered for this list, but excluded because of a high item response rate.

Missing values for the variables measuring student ability in mathematics (theta), the associated standard error of theta (sem), and socioeconomic status (SES) were dealt with using multiple imputations to produce five estimated values for each variable. For all variables with imputed values, indicator variables (flags) were created to allow users to easily identify which cases had been imputed.

To alleviate the problem of missing data from a respondent record for first follow–up, statistical imputation methods were employed similar to those used for the HSLS:09 base year. More specifically, a weighted sequential hot–deck imputation procedure using the final student analysis weight was applied to the missing values. Four key analysis variables were identified for single–value imputation from the edited HSLS:09 first follow–up data. Additional variables were considered for this list but were excluded because of either high item response rate or they were deemed to be of little analytic importance.

To alleviate the problem of missing data from a respondent record for the second follow-up, statistical imputation methods were employed for the second follow-up that were similar to those used for the HSLS:09 base year, first follow-up, and 2013 Update. Ten key analysis variables were identified for single-value imputation from the second follow-up data. Stochastic methods were used to impute the missing values. Specifically, a weighted sequential hot-deck (WSHD; statistical) imputation procedure using the final second follow-up student analysis weight (W4STUDENT) was applied to the missing values for the ten variables. The WSHD procedure replaces missing data with valid data from a donor record (i.e., item respondent) within an imputation class. In general, variables with lower item nonresponse rates were imputed earlier in the process. Regardless of the method, indicator variables (flags) were created to allow users to easily identify the imputed values.

Top

Future Plans

The number and timing of future follow-ups beyond 2016 is yet to be determined, although the expectation is that the cohort will be followed at least to age 30, with a questionnaire administration and a postsecondary education transcript collection in 2025-26.

Data Comparability

Comparability with Questionnaire Data in Other NCES Secondary Longitudinal Studies. The HSLS:09 data do not directly support certain cross–cohort comparisons that were possible in earlier NCES secondary longitudinal studies. In earlier secondary longitudinal studies, comparisons were possible because each cohort was similarly defined and because, by design, a core set of questions had been repeated across studies. However, students in HSLS:09 were fall–term 9th–graders in the base year, and in the first follow–up were spring–term 11th–graders, which does not correspond to the prior studies’ cohorts (spring–term 8th–, 10th–, or 12th–graders). Therefore, HSLS:09 does not allow for an intercohort time–lag study.

Nonetheless, comparisons can be made in a couple of ways: (1) coursetaking can be compared between HSLS:09 and ELS:2002, NELS:88, and HS&B, based on the continuous data for grades 9 through 12 that are supplied by high school transcripts; and (2) because HSLS:09 models the same transition–from adolescence in the high school years to young adulthood, as marked by educational attainment, work and career, and family formation–the design answers the same basic questions as the predecessor studies. Moreover, all of the studies have essentially the same sampling designs, provide nationally representative data across public and private schools, and define race/ethnicity domains similarly across cohorts. Thus, while each longitudinal study may have slight differences in emphasis, all draw content from the same or similar theoretical constructs (e.g., achievement growth, school effectiveness, social capital, social attainment, human capital).

Comparability with Student Assessment Data in Other NCES Studies. Differences in the content and scaling of the HSLS:09 academic assessment (i.e., algebraic reasoning) and tests administered in prior NCES secondary longitudinal studies severely limit the possibility of comparisons. Moreover, apart from a handful of National Assessment of Educational Progress (NAEP) items, there are no common items that link the HSLS:09 test to earlier mathematics assessments, and due to the testing points—fall of 9th grade and spring of 11th grade—the assessment results are not comparable to prior studies, such as the Program for International Student Assessment (PISA) or NAEP. Therefore, even a weak linkage, such as a concordance, would seem inadvisable to implement.

New Features of HSLS:09. Some of the new, distinctive, and innovative features of HSLS:09, compared to the previous NCES secondary longitudinal studies, include the following:

• use of a computer-administered assessment and student questionnaire in a school setting; 
• an assessment that focuses on algebraic reasoning; 
• use of computerized (web/CATI) parent, teacher, administrator, and counselor questionnaires; 
• inclusion of a school counselor survey;
• starting point in the fall of ninth grade; 
• emphasis on the dynamics of educational and occupational decision making; and 
• enhanced emphasis on STEM trajectories;
• in first follow-up, questionnaire and assessment also computer-administered for out-of-school and transfer students.

Although the first follow-up data in particular are designed to facilitate the analysis of change, including gain in mathematical proficiency, and its correlates, the data cannot be used cross–sectionally; unlike the base year, the first follow–up data cannot be used cross–sectionally because freshening for an 11th–grade cohort was not conducted. Given a 2009 ninth–grade cohort 2.5 years later, first follow–up data can only be used longitudinally.

Top

Top