​​​​​​​​​​​NAEP Technical DocumentationSampling of Public Schools for the 2022 Long-Term Trend Assessment

In designing the public school long-term trend sample for each age, seven objectives underlie the process of determining the probability of selection for each school and the number of students to be sampled from each selected school:

• to meet the overall target student sample size;
• to select an equal-probability sample of students from the age population;
• to limit the number of students selected from any one school;
• to ensure that the sample within a school does not include a very high percentage of the students in the school, unless all students are included;
• to reduce the sampling rate of small schools, in recognition of the greater cost and burden per student of conducting assessments in such schools;
  
• to increase the number of American Indian/Alaska Native (AI/AN), Black, and Hispanic students in the sample; and
  
• to ensure the inclusion of all eligible schools that were part of the 2020 public school long-term trend sample for each age, respectively.
  

The goal in determining the school's measure of size is to optimize across the second to the fifth objectives in terms of maintaining the precision of estimates and the cost effectiveness of the sample design.

Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the next four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated age enrollment as indicated on the sampling frame.

The measures of size vary by enrollment size. The initial measures of size, \(MOS_{js}\), were set as follows: \begin{equation} MOS_{js} = PSU\_WT_{s} \times \left\{\begin{array}{llll} x_{js}{,} & \text{if } z_{js} < x_{js} \\ y_{j}{,} & \text{if } 19 < x_{js} \leq z_{js}\\ \biggl(\dfrac{y_j}{20}\biggr) \times x_{js}{,} & \text{if } 10 < x_{js} \leq 19 \\ \dfrac{y_j}{2}{,} & \text{if } x_{js} \leq 10 \\ \end{array}\right. \end{equation}

where \(PSU\_WT_{s}\) is the PSU weight (i.e., the inverse of the PSU probability of selection) for school \(s\); \(x_{js}\) is the estimated age enrollment for school \(s\) for sample age \(j\); \(y_{j}\) is the target within-school student sample size for sample age \(j\); and \(z_{js}\) is the within-school take-all student cutoff for school \(s\) for sample age \(j\). The target within-school sample size and the within-school take-all cutoff were both 50.

The measures of size for schools in the Honolulu primary sampling unit (PSU) were doubled to increase their chances of selection. Schools in the Honolulu PSU have their measures of size doubled to ensure at least one sampled school from the PSU. The Honolulu PSU is a certainty not due to its size, but because it is unique ​due to its high population of Asian and Native Hawaiian/Pacific Islander students. The preliminary measures of size \(M_{js}\) for schools in the Honolulu PSU were set as \begin{equation} M_{js} = 2 \times MOS_{js}. \end{equation}

Preliminary measures of size were set equal to the initial measures of size for schools whose measures of size were not doubled or quadrupled.

The preliminary school measure of size is rescaled to create an expected number of hits by applying a multiplicative constant \(b_{j}\), which varies by age \(j\). One can choose a value of \(b_{j}\) such that the expected overall student sample yield matches the desired target specified by the design, where the expected yield is calculated by summing the product of an individual school's probability and its student yield across all schools in the frame.​​

The final measure of size, \(E_{js}\), is defined as

\begin{equation} E_{js}=min(b_{j}\times M_{js},u_{j}). \end{equation}

The quantity \(u_{j}\) (the maximum number of hits allowed) in this formula is designed to put an upper bound on the burden for the sampled schools. For public schools, \(u_{j}\) is 1 because by design a school could not be selected, or hit in the sampling process more than once for the given sample age.

In addition, new and newly-eligible schools were sampled from the new school frame. The final measure of size for these schools, \(E_{js}\) is defined as

\begin{equation} E_{js}=min(b_{j}\times M_{js} \times \pi_{djs}^{-1},u_{j}). \end{equation}

The variable \(\pi_{djs}\) is the probability of selection of the district \(d\) into the new-school district sample.

To address the objective in the last bullet above, an adjustment was made to the initial measures of size in an attempt to ensure the inclusion of all eligible schools that were part of the 2020 public school long-term trend sample for each age. The NAEP sampling procedures used an adaptation of the Keyfitz process to compute conditional measures of size that, by design, maximized the overlap of schools selected for both the 2020 and 2022 long-term trend assessments.

Schools were ordered within each jurisdiction using the serpentine sort described under the stratification of public schools. A systematic sample​ was then drawn using this serpentine-sorted list and the measures of size. The numbers of public schools selected were approximately 410 for age 9 and 480 for age 13. 

Printer-friendly Version