NAEP Technical DocumentationNew-School Sampling Frame for the 2018 Public School Social Sciences Assessment

The most current Common Core of Data (CCD) file available was used to construct the public school frame for NAEP 2018. However, the information on that file was two years out of date by the time of the NAEP assessment. During that two-year period, some schools closed, others changed grade span, and still others came into existence.

One can improve coverage by asking districts to provide information on currently open schools that were not listed in the 2015–2016 CCD file used to create the NAEP public school frame, and also to report grade span changes that may have caused a CCD-listed school to become newly eligible for eighth grade. Asking all districts to do this would have imposed an undue burden, so instead, a random sample of districts was contacted to obtain lists of new and newly eligible schools. The goal was to allow every new or newly eligible school a chance of selection, thereby fully covering the target population of schools in operation during the 2017–2018 school year.

The first step in this process was the development of a new-school frame through the construction of a district-level file from the CCD school-level file. The new-school frames for both social sciences and technical engineering literacy (TEL) were constructed at the same time. Since the social sciences and TEL assessments were to be conducted within a total of 138 primary sampling units (PSUs), only districts that fell within the boundaries of those PSUs were eligible for sampling. Once the district-level file was subset to just the targeted PSUs, it was divided into three files: the first containing state-operated and charter school districts, the second containing small districts, and the third containing large districts.

State-operated districts and districts containing no schools other than charter schools require special handling. In survey years when state-level assessments are conducted, NAEP State Coordinators are asked to provide the names of all new charter-only and state-run schools. However, these types of school districts tend not to be geographically compact, and it is not feasible to link such a district to a single PSU, except at the individual school level. The smaller the proportion of a state’s population falling within sampled PSUs, the less likely that a specific new school of this type will be added to the frame and the more likely that state personnel will have expended unnecessary effort in providing updated information that will not be used. For this reason, for the NAEP 2018 assessment, the charter-only and state-run district component of the new school procedure was implemented only in states where more than 60 percent of youth fell within sampled PSUs. This meant that this component of the new-school sampling frame procedure was implemented in 21 states plus the District of Columbia, which taken together contain about 67 percent of the nation's youth.

The remaining districts were classified as small or large. A small district usually contains no more than three schools on the frame in total, with no more than one school at each targeted grade (fourth, eighth, and twelfth). However, for NAEP 2018 new schools were only selected for grade 8 assessments. Therefore, for NAEP 2018, a small district contains no more than one school with grade 8 on the frame in total. New schools in small districts were identified during school recruitment and added to the sample if the frame school in the same district was sampled for eighth grade. From a sampling perspective, the new school was viewed as an “annex” to the sampled school that had a well-defined probability of selection equal to that of the frame school. Thus when the frame school was sampled in a small district for eighth grade, any new school was automatically sampled for eighth grade as well.

Large districts were divided into 77 strata based on the NAEP 2018 PSU sampling strata, with districts in certainty PSUs grouped together in a single stratum. The district sample was allocated to each of the 77 strata proportional to the percent of the U.S. population of eighth-graders contained in that stratum, with the caveat that each stratum had to be allocated at least one district. This allocation was then adjusted because it resulted in too many districts in the certainty strata and not enough in the noncertainty strata. Once the allocation to each stratum had been fixed, districts were sampled from a sorted list using systematic sampling with probability proportional to size and a random start, with the measure of size being the number of eighth-graders enrolled in the district. Within the certainty PSU stratum, districts were sorted in a serpentine manner by state and measure of size prior to sampling. In all other strata the districts were sorted by measure of size alone. District selection probabilities were retained and used in all subsequent stages of sampling and weighting.

The selected districts were then sent a listing of all their schools that appeared on the 2015–2016 CCD file and were asked to provide information about any schools missing from CCD, and grade span changes of existing schools. This information provided by the sampled districts was used to construct sampling frames for the selection of new or newly eligible public schools and also for updating the status of existing schools (e.g., school closings). This process was conducted through the NAEP State Coordinator in each jurisdiction. The coordinators were sent the information for all sampled districts in their respective states and were responsible for returning the completed updates.

The eligibility of a school was determined based on the grade span and whether it was located in a sampled PSU. A school was also classified as "newly eligible" if a change of grade span had occurred such that the school status changed from ineligible to eligible at eighth grade.

This process yielded 307 schools on the eighth-grade new school sampling frame for social sciences. These schools contained an estimated 19,143 eighth-grade students.

Printer-friendly Version