High School Longitudinal Study of 2009 (HSLS:09)

4. SURVEY DESIGN

TARGET POPULATION

The target population at the school level was defined as regular public schools, including public charter schools, and private schools, in the 50 states and the District of Columbia, providing instruction in both 9th and 11th grade. The target population of students was defined to include all ninth–grade students who attended the study–eligible schools in the fall 2009 term. Public schools, including public charter schools, and private schools in the 50 states and the District of Columbia providing instruction to both 9th–grade and 11th–grade students, were sampled in the base year of HSLS:09.

SAMPLE DESIGN

In the fall of 2009, ninth–graders were sampled within selected schools. All ninth–grade students in the sampled schools were classified as eligible for the study, including students with disabilities and English language learners who may not have been capable of completing the survey instruments. Moreover, the base–year dataset is not only nationally representative of ninth–graders in schools with both 9th and 11th grades, but also includes 10 individual state–level representative samples of students and schools. The first follow–up sample consisted of those students selected for the base year in 2009–10 that are still eligible for HSLS:09.

Base–year survey. In the base–year of HSLS:09, students were sampled through a two–stage process. First, stratified random sampling and school recruitment resulted in the identification and contacting of 1,889 eligible schools. The primary sample of regular public and public charter schools was selected from the 2005–06 Common Core of Data (CCD). Private schools were sampled from the 2005–06 Private School Universe Survey (PSS).

The following is a complete list of criteria used to exclude schools from the sampling frame: Bureau of Indian Education (BIE), special education, career technical education, Department of Defense schools located outside the United States, and ungraded schools, as well as schools not in operation during the fall of 2009, schools without both 9th and 11th grades, juvenile correction facilities, schools that only offer testing services for home–schooled students, and schools that do not require students to attend daily classes at their facility.

The national design called for the selection of a sufficient sample to yield 800 eligible, participating schools–600 public schools and 200 private schools–which represented a similar proportion of each school control type in the population. However, the design also called for the oversampling of Catholic schools relative to other types of private schools; thus, 100 Catholic schools were chosen (or 8 percent of all eligible Catholic schools), and 100 other private schools were chosen (2 percent of eligible other private schools). The overall school sample size was allocated to the sampling strata in proportion to the relative number of ninth–grade students within the strata. A total of 48 mutually exclusive first–stage sampling strata were created by cross–classification of three variables: school type or sector (public, private–Catholic, private–other); region of the United States (Northeast, Midwest, South, West); and locale (city, suburban, town, rural). A sample of 1,889 eligible schools were selected, and about 940 schools eventually participated in the first wave of HSLS:09.

In the second stage of sampling, students were randomly sampled from school ninth–grade enrollment rosters, with 25,206 eligible selections (or about 28 students per school). A stratified systematic sample was drawn from the enrollment lists where the strata were equivalent to four categories of race/ethnicity–Hispanic, Asian, Black, and Other with inflated overall sampling rates for Asian students to ensure sufficient size for analysis. All students who met the target population definition were deemed eligible for the study. However, not all students were capable of completing a questionnaire or assessment. Students who, due to language barriers or severe disabilities, were unable to directly participate in the study were retained in the sample, and contextual data were sought for them. (Their ability to complete the study instruments was reassessed in the first follow-up in 2012.) Of the 25,210 eligible students, 550 were classified as questionnaire-incapable due to physical limitations, cognitive disabilities, or limited English proficiency, and an additional 3,210 were nonrespondents.

First follow–up survey. The first follow–up target populations are the same as defined for the base year. Consequently, the student target population contains all 9th–grade students as of fall 2009 who attended either regular public or private schools, in the 50 United States and the District of Columbia, that provide instruction in both 9th and 11th grade. This population is referred to as the ninth–grade cohort.

All of the 944 base–year participating schools were eligible for the HSLS:09 first follow–up. No new sample of schools was selected for this round. Therefore, the base–year school sample in the first follow–up is not representative of high schools with 9th and 11th grades in the 2011–12 school year, but rather is intended as an extension of the base–year student record, to be used to analyze school–level effects on longitudinal student outcomes.

All 25,206 base–year study–eligible students, regardless of their response status, were included in the first follow–up sample. Unlike prior NCES studies, the HSLS:09 student sample was not freshened to include a representative later–grade cohort (such as 11th–graders in HSLS:09) as was done with 12th–graders in the Education Longitudinal Study of 2002, for example. Therefore, first follow–up estimates from the sample are associated only with the 9th–grade cohort 2.5 years later, and not the universe of students attending the 11th grade in the spring of 2012.

Explaining changes in estimates from the base year to the first follow-up is of prime importance to researchers interested in HSLS:09. To ensure sufficient resources to maximize response from the sampled students, a decision was made to select a random subsample of parents in the first follow-up, with the goal of achieving 7,500 or more parent interviews.

The subsample of parents was randomly selected from within categories defined by the combination of the base year first– and second–stage sampling strata. The parent subsample was selected using a PPS minimal replacement methodology and the student base weight as the measure of size. Use of the base weight from the base year minimized the variation in the first follow–up student home–life contextual base weights. This sampling approach has been used in other NCES surveys such as the National Education Longitudinal Study of 1988 fourth follow-up to subsample prior–wave nonrespondents.

2013 Update and High School Transcript Collection. Of the 25,206 students eligible for the base year, 25,168 were eligible for the 2013 Update and the High School Transcript Study (a total of 38 were deceased). Not all cases were fielded: sample members were excluded from fielding when neither base–year nor first follow–up data were collected for them, or were out of scope for a given round in accordance with one of four out of scope categories: incapable of meaningful participation, inaccessible, deceased, or study withdrawal. These unfielded cases are classified as nonrespondents and appear in the sample denominator for calculation of response rates.

Second follow-up survey. The second follow-up fielded sample included 23,316 of the 23,401 sample members fielded and found eligible for the 2013 Update. The 85 sample members not fielded withdrew from the study between the end of the 2013 Update collection and the beginning of the second follow-up data collection or were found to be deceased.

Top

Data Collection and Processing

Reference dates. In the base–year survey, recruitment of school districts and schools began a year before data collection activities commenced. In–school data collection (from September 2009 through February 2010) comprised a student questionnaire and an assessment of algebraic reasoning. Students who did not participate in the initial in–school session were contacted to complete the questionnaire outside of school. Out–of–school data collection (from September 2009 through May 2010) comprised parent and school staff (school administrator, teacher, and school counselor) questionnaires.

The first follow–up of HSLS:09 took place in 2012 when most of the cohort were in the second semester of their 11th–grade school year. The first follow–up assessment was administered in two settings: in–school (as in the base year) and out–of–school in a self–administered web–based environment. The 2013 Update occurred in summer and fall 2013, when most members had already graduated from high school. The second follow-up data collection ended with a 68 percent weighted response rate.

Data collection. Prerecruitment activities for school districts and schools began with the solicitation of study endorsements (HSLS:09 was endorsed by 30 organizations) and a courtesy notification to the states. Obtaining cooperation from school districts, dioceses, and schools followed. Once schools agreed to participate, the recruitment team worked with them to set up study logistics for the student sessions and to facilitate list collection.

School recruitment. Before school recruitment began, the Chief State School Officer (CSSO) from each state was notified that HSLS:09 would be conducted in districts and schools in his or her state. No follow–up was performed at the state level. Recruitment commenced with public school districts, and information packages were sent to the superintendent of each district and diocese containing sampled schools. For public and Catholic schools, school–level contact commenced upon receipt of district or diocesan approval. The sampled non–Catholic private schools were contacted directly because it was not necessary to wait for higher approval. For these schools, the principal received an informational package and later was contacted by the recruiting team to answer any questions about the study and to provide an overview of the various data collection activities.

An exception to this recruitment procedure occurred for sampled school districts and public schools in 10 states that were identified for an augmentation (supported by the National Science Foundation) to allow for the collection of data that would be representative at the state level. (Information on the 10 states selected is documented in materials available for restricted–use data license holders.) If any of the 10 states had not already sampled enough public schools to generate representative state–level data with a reasonable level of precision (ideally, 40 or more participating schools), additional schools were contacted in order to achieve the desired yield.

For each school selected to participate in HSLS:09, upon gaining access, recruiters identified a school coordinator to serve as a point of contact and to provide logistical information. The school coordinator was responsible for scheduling the in–school sessions for data collection and identifying the appropriate staff members to complete the school administrator questionnaire and school counselor questionnaire. The school coordinator was also responsible for working with school personnel to specify the type of parental permission required for the in-school student sessions and to grant permission to use Sojourn (Linux operating system) on the school’s computers.

For the first follow–up, 5 of the 944 schools were found to be closed or had no eligible sampled students still enrolled in the base–year school. Of the eligible 939 schools, 904 base–year schools (96 percent) agreed to continue participation in the HSLS:09 first follow–up.

Student data collection. Student data collection was conducted in 944 high schools by trained session administrators. Student sessions were composed of a computerized questionnaire and an assessment of algebraic reasoning. The session administrator and school coordinator distributed the permission forms and tracked their return, confirmed the eligibility and capability of sampled students, and determined whether any sampled students needed special accommodations to participate in the study. Students were deemed incapable of participating if they had a physical or cognitive disability or a language barrier that precluded them from participating in the base–year data collection.

HSLS:09 first follow–up student questionnaires were completed in one of four data collection modes: in–school, web, CATI, and CAPI. The student questionnaire was completed by 82 percent of eligible sampled students in the first follow–up. Sixty–one percent of students completed the questionnaire in school, while 20 percent completed the questionnaire outside of school, which comprised students who were no longer enrolled in the base–year school and those who missed the in–school session. During out–of–school data collection, 9 percent of student respondents completed the questionnaire via the web, 6 percent completed the questionnaire with a field interviewer, and 5 percent completed the questionnaire by phone.

The second follow-up was conducted from March 2016 through January 2017, approximately 3 years after high school graduation for most of the cohort.

Parent data collection. One parent of each sampled student was asked to complete a 30–minute questionnaire. The parent questionnaire could be self–administered on the web or completed with a professional interviewer via computer–assisted telephone interviewing (CATI). Additionally, to reduce nonresponse, a brief paper-and-pencil questionnaire containing critical items was sent to nonresponding parents near the end of data collection.

In response to a lower–than–desired response rate to the parent questionnaire, an incentive experiment was implemented about 3 weeks prior to the end of data collection. Parents were included in the experiment if one of three criteria was true: (1) the sample member refused to participate but was not coded a final refusal; (2) 15 or more calls had been placed to the sample member, or (3) the sample member had an address but no phone number was found after all intensive tracing processes had been exhausted. At 47 percent, the highest percentage of completed interviews was achieved by parents who were offered $20 (as opposed to $10 or $0) and who had been included in the experiment because they have received more than 15 CATI calls.

Among the subsample of parents contacted to participate in the HSLS:09 first follow–up, about 72 percent completed a questionnaire. The average time to complete a parent questionnaire across all data collection modes was 37 minutes. Time to complete the parent questionnaire varied by mode with web respondents averaging 34 minutes, CAPI respondents averaging 37 minutes, and CATI respondents averaging 40 minutes.

School staff data collection. In addition to the student and parent questionnaires, the school administrator, a school counselor, and the math and science teachers of each sampled student were asked to complete a 30–minute questionnaire. Each staff questionnaire was available on the web or via CATI.

Like the base–year data collection, contacting of school districts and schools for first follow–up began a year before data collection commenced. In–school data collection comprised a student questionnaire and mathematics assessment. Students who did not participate in the in–school session, including those who were no longer enrolled at the base–year school, were contacted to complete the questionnaire and mathematics assessment outside of school. First follow–up data collection also included surveys of school administrators, counselors, and a subsample of parents. There was no teacher data collection in the first follow–up.

Top

Data Processing. All questionnaire data were stored in an SQL server database. CATI applications were used to obtain participation where web interviews could not be obtained; however, the data were stored in the same SQL server database. SQL data were exported nightly into SAS datasets. Cleaning programs were designed to partition the data into questionnaire datasets and methodological datasets and to attach variable names and labels.

All respondent records in the final dataset were verified with the case management/control system to identify inconsistencies. For example, it was possible that data were collected from a sample member who later was set to nonrespondent status. It would not be appropriate to retain these data, and the case management/control system served as a safeguard to ensure they were removed.

Documentation procedures were developed to capture variable and value labels for each item. Item wording for each question was also provided as part of the documentation. This information was loaded into a documentation database that could export final data file layouts and format statements used to produce formatted frequencies for review. The documentation database also had tools to produce final electronic codebook input files.

For each type of questionnaire (e.g., student, parent, and school administrator), the survey instrument was the same regardless of data collection mode (web survey and CATI). Responses for each type of questionnaire were thus able to be stored in a SQL server database regardless of the collection mode used. This helped ensure that skip patterns were consistent across collections. An exception to this standard was for parent data, since an abbreviated paper–and–pencil instrument was administered for nonresponse conversion. The abbreviated parent questionnaire was designed to include key questions from the instrument that could be entered into the parent questionnaire database.

Data editing. Editing programs were developed to identify and output inconsistent items across logical patterns within questionnaires. These items were reviewed, and rules were written to correct previously answered (or unanswered) questions to stay consistent with previously answered items.

Programs were also developed to review for consistencies across multiple sources of data and identify discrepancies that required further review and resolution. For example, the student’s sex was obtained from the school and stored in his or her roster data; in addition, the student’s sex was collected in the student interview and the parent interview. If there was a discrepancy across sources, the student’s first name was reviewed to determine and store the correct value.

For first year follow–up, consistency checks were included for unlikely patterns across rounds (i.e., between base year and first follow–up) as well as across sources within a given round (e.g., between parent and student reports). Additionally, the HSLS:09 first follow-up parent instrument included tools that allowed online coding of literal responses of occupation job title and duties to the 2000 Standard Occupational Classification (SOC) taxonomy. The HSLS:09 first follow–up student instrument also asked respondents to indicate what occupation they thought they would have when they were age 30. Students entered a job title, but were not asked to enter job duties. Respondents also had the option of checking a box to indicate that they did not know. Students were not asked to code their expected occupations so all job titles needed to be coded after data collection using the O*NET taxonomy. The text strings were first matched against coded strings from the base year. When text strings matched between base year and first follow–up, the base–year code was applied to the first follow–up text string.

The following editing steps were implemented: 

• rule–based edits (i.e., changes that were made based on patterns in the data); 
• hard–coded edits based on changes recommended by a reviewer if respondents misunderstood the questionnaire (e.g., respondent was instructed to enter a percentage; however, there was strong evidence that the respondent entered a count rather than the percentage); and
• edits based on logical patterns in the questionnaire (e.g., skip pattern relationships between gate and dependent questions) .  

Top

Estimation Methods

Weighting. Analytic weights are used in combination with software to account for the complex survey design of HSLS:09 and produce estimates that are nationally representative, with appropriate standard errors. The HSLS:09 base–year contains five sets of analytic weights: a school–level weight; a student–level weight; and three special student–level weights: two linked with contextual data from science and mathematics courses and one linked with parent–reported family and home contextual data.

The school–level weight can be used for school–level analyses involving the school administrator and counselor questionnaires. The student–level weight is for student–level analyses using student response data. In contrast, because of the low unit response rates for parents and teachers, the three special student–level weights are used for analyses at the student level that rely on a combination of student, parent, and teacher response data. Importantly for such analyses, the student still serves as the unit of analysis, and the parent and teacher data are used to provide contextual information. Corresponding balanced repeated replication (BRR) weights were constructed in a similar fashion as the analytic weights and should be used to achieve proper variance estimates.

The first follow–up data file contains a total of nine analytic weights: five weights for analysis of the base–year data and four weights to be used in conjunction with the first follow–up data (two weights for analysis of first follow–up responses, and two weights for analysis of population change from base year to first follow–up). In summary, researchers analyzing any data from the first follow–up (alone or in conjunction with base–year data) should use one of the four first follow–up weights. Analyses involving only the base–year data, with no first follow–up data, should include one of the five weights for analysis of base–year data. Three sets of weights were created on the cumulative analytic first follow–up file: a set of base–year student BRR weights; a set of first follow–up student BRR weights, and a set of base–year to first–follow–up longitudinal weights.

Two sources of contextual information for analysis of the student data were obtained in the HSLS:09 base year but not in the first follow–up. They include interviews with the science teacher and mathematics teacher for students taking the associated course in the ninth grade. Researchers may choose to condition the analyses of first follow-up student data on teacher responses obtained in the base year. Unlike the base-year data file, the HSLS:09 first follow–up data file does not contain contextual analytic weights to account for nonresponse among students with base-year teacher information. Instead, either student or parent weights should be used depending on the inclusion of parent responses. Note that estimates generated with student data and either the student or parent weight, in conjunction with the base–year teacher responses, are no longer associated with the HSLS:09 target population of ninth–grade students and should be used with caution.

School level. The elements combined to form the school analytic weight are a base weight, two nonresponse adjustments, and a final calibration adjustment.

An initial base weight (sometimes referred to as a design or sampling weight) was constructed as the inverse of the probability of selection. Then, the base weight was adjusted for (1) school administrators who declined to participate in HSLS:09, but provided information as part of the nonresponding–school questionnaire; and (2) school administrators who declined to participate in HSLS:09 and did not provide information for the nonresponding-school questionnaire. Both adjustment factors were constrained to minimize excess variation in the resulting weight. A final adjustment was applied to school weights to calibrate the sum of the analytic weights to target population counts tabulated from the 2007–08 CCD and the 2007–08 PSS. The calibration adjustments are also known to reduce coverage bias and variation in the resulting analytic weights, improving precision in the survey estimates.

Student level. The components of the student analytic weights are a base weight, two nonresponse adjustments, and a final calibration adjustment.

HSLS:09 ninth–grade students were randomly selected from four race/ethnicity sampling strata (Hispanic, Asian, Black, and other). The conditional base weight for students in each of the race/ethnicity strata was constructed as the inverse of the probability of selection within the school sampled in the first stage of the design. Though the weighted response rate was above the 85 percent threshold, a nonresponse adjustment weight was developed to address two sources of bias: parent refusal to give permission to participate in the study and student refusal to participate.

There were 24,660 questionnaire–capable students in the sample. Approximately 9 percent (n = 2,380) did not participate because of a parent refusal. To minimize bias associated with this type of student nonresponse, a nonresponse adjustment was applied to the weights of the 22,280 questionnaire–capable students without a parent refusal. Note that the decision of the student to participate in the study was determined prior to data collection. Thus, all nonresponding students were classified as questionnaire capable, and the questionnaire–incapable students were excluded from the weight adjustment.

The sum of the nonresponse-adjusted weights was compared against totals tabulated from the 2007–08 NCES sampling frame files of eligible schools. The weighted sums were less than the sampling frame counts; therefore, a calibration adjustment was applied so that the weighted sums matched the estimates from the sampling frame.

Student linked with science and mathematics course weights. Teacher background and limited classroom information was collected from the science and mathematics teachers of sampled students during the fall of 2009. Weighted response rates for science and mathematics teachers were 70 and 72 percent, respectively. Nonresponding teachers were linked with 32 percent of the science enrollees and 25 percent of the mathematics enrollees. To account for the loss of student records resulting from nonresponding teachers, two subject–specific enrollee weights were created for student–level analyses that used classroom context information. The two weights were independently created by adjusting the main student analytic weight.

Typically, variables used for a nonresponse weight adjustment are only effective if they are related to the response patterns exhibited in the data. However, since teachers in HSLS:09 were not sampled directly, information was not available on nonresponding teachers. Consequently, a weight adjustment could not be calculated to adjust for patterns of HSLS:09 teacher nonresponse. Instead, students linked to a responding teacher were combined with students not enrolled in the course and then the weights were calibrated to the sum of the final student analytic weight for the full set of course enrollees.

Student linked with parent-reported family and home life weights. Information on factors affecting family life and background, as well as parent/guardian opinions on education and school involvement, were collected through the parent questionnaire. The weighted parent/guardian response rate was 68 percent. As with the adjustments for the weights used in analyses involving data from science and mathematics teachers, information on nonresponding parents was not available; therefore, adjustments to weights for parental nonresponse relied on using student data to calibrate the final student analytic weight.

 

Top