​​​​​​​​​​​NAEP Technical DocumentationComputation of Measures of Size for the 2019 Fourth- and Eighth-Grade Public School National Science Assessment

In designing the public school science assessment samples, six objectives underlie the process of determining the probability of selection for each school and the number of students to be sampled from each selected school:

• to meet the overall target student sample size;
• to select an equal-probability sample of students from each sampling stratum;
• to limit the number of students selected from any one school;
• to ensure that the sample within a school does not include a very high percentage of the students in the school, unless all students are included;
• to reduce the rate of sampling of small schools, in recognition of the greater cost and burden per student of conducting assessments in such schools; and
• to increase the number of American Indian/Alaskan Native (AIAN), Black, and Hispanic students in the sample.

The goal in determining the school's measure of size (MOS) is to optimize across the middle four objectives in terms of maintaining the precision of estimates and the cost effectiveness of the sample design.

Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the next four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated grade enrollment as indi​​​cated on the sampling frame.

\begin{equation} MOS_{js} = \left\{\begin{array}{llll} x_{js} & \text{if } z_{js} < x_{js} \\[2pt] y_{j} & \text{if } 20 < x_{js} \leq{z_{js}} \\ \biggl(\dfrac{y_j}{20}\biggr) \times x_{js} & \text{if } 5 < x_{js} \leq {20} \\ \dfrac{y_j}{4} & x_{js} \leq {5} \end{array}\right. \end{equation}

where \(x_{js}\) is the estimated grade enrollment for school \(s\) in stratum \(j\),  \(y_{j}\) is the target within-school student sample size for stratum \(j\), and `z_{js}` is the within-school take-all student cutoff for stratum `j` to which school `s` belongs. ​​​​

To increase the number of AIAN students in the sample, the measures of size for schools with relatively high proportions of AIAN students (5 percent or more and with at least 5 AIAN students) were quadrupled.

Likewise, to increase the number of Black and Hispanic students in the sample, the measures of size for schools with relatively high proportions of Black and Hispanic students (15 percent or more and with at least 10 Black or Hispanic students) were doubled.

This approach is effective in increasing the sample sizes of AIAN, Black, and Hispanic students without inducing undesirably large design effects on the sample, either overall, or for particular subgroups.

Colorado, Hawaii, Kansas, Maryland, Ohio, and South Dakota were excluded from minority oversampling because these states had indicated that they would not participate in the science assessment prior to sampling. This was done to avoid potentially increasing sample in states that were unwilling to participate in this assessment.

For schools with high proportions of AIAN students, the preliminary measures of size were calculated as follows:

\begin{equation} MOS_{js} = 4 \times \left\{ \begin{array}{l} x_{js} & \text{if } z_{js} < x_{js} \\[2pt] y_{j} & \text{if } 20 < x_{js} \leq{z_{js}} \\ \biggl(\dfrac{y_j}{20}\biggr) \times x_{js} & \text{if } 5 < x_{js} \leq {20} \\ \dfrac{y_j}{4} & x_{js} \leq {5} \end{array}\right. \end{equation}

For schools with high proportions of Black and Hispanic students, the preliminary measures of size were calculated as follows:

\begin{equation} MOS_{js} = 2 \times \left\{ \begin{array}{l} x_{js} & \text{if } z_{js} < x_{js} \\[2pt] y_{j} & \text{if } 20 < x_{js} \leq{z_{js}} \\ \biggl(\dfrac{y_j}{20}\biggr) \times x_{js} & \text{if } 5 < x_{js} \leq {20} \\ \dfrac{y_j}{4} & x_{js} \leq {5} \end{array}\right. \end{equation}

For all other schools (those with low proportions of AIAN and Black and Hispanic students and in the six nonparticipating states), the preliminary measures of size were calculated as follows:

It follows that the final measure of size, \(E_{js}\), was defined as: $$E_{js}=min(b_{j}\times MOS_{js},u_{j})$$ where \(u_{j}\) is the maximum number of hits allowed. For the 2019 public school science samples, the limit was one hit.

One can choose a value of \(b_{j}\) such that the expected overall student sample yield matches the desired target specified by the design, where the expected yield is calculated by summing the product of an individual school's probability and its student yield across all schools in the frame.

The school's probability of selection, `\pi_{js}`, was given by

\begin{equation}\pi_{js}=min(E_{js},1).\end{equation}

In addition, new and newly-eligible schools were sampled from the new-school frame. The assigned measures of size for these schools, $$E_{js}=min(b_{j}\times MOS_{js} \times \pi_{djs}^{-1},u_{j})$$ used the \(b_{j}\) and \(u_{j}\) values from the CCD-based school frame for stratum \(j\) (i.e., the same sampling rate as for the CCD-based school sample within each stratum). The variable \(\pi_{djs}\) is the probability of selection of the district into the new-school district (\(d\)) sample.

In addition, an adjustment was made to the initial measures of size in an attempt to reduce school burden by minimizing the number of schools selected for both the state mathematic and reading assessments and the national science assessments in public schools. The NAEP sampling procedures used an adaptation of the Keyfitz process to compute conditional measures of size that, by design, minimized the overlap of schools selected for both the state assessments and national assessments.

Printer-friendly Version