​​​​​​​​​​NAEP Technical DocumentationEvaluation of the Samples for the 2022 State Assessment Using State Achievement Data

The purpose of this analysis was to determine whether public schools selected for the 2022 samples were representative of the schools on the NAEP sampling frames in terms of student achievement. Percentiles of the achievement distributions were compared between the frame and school sample for each public school jurisdiction in grades 4 and 8.

Achievement Data

For grades 4 and 8, the achievement variable used in the analysis was the same variable used in the NAEP sample design to stratify the public school frame. For all jurisdictions in the analysis except Puerto Rico, the variable was an achievement score provided by the jurisdiction. However, for Puerto Rico, where achievement data were not available, the 2015–2019 American Community Survey (ACS) 5-year estimates for median household income were used. (Median household income was based on the five-digit zip code area in which the school was located.) The achievement data consisted of various types of school-specific achievement measures from state assessment programs. The type of achievement data available varied by jurisdiction. For instance, in some states, the measure was the average score for a given state assessment. In other states, the measure was a percentile rank or percentage of students above a specific score. For Connecticut at grade 4, for example, we used the percentage of students in grade 4 who scored at or above the proficient level on the state mathematics test.

During frame development, not every record on the Common Core of Data (CCD)​ file matched the achievement data files created for the National Center for Education Statistics (NCES), even in jurisdictions where those data were generally available. For schools that did not match, their achievement scores were imputed by a mean matching imputation approach using the mean achievement score for schools with complete achievement data within the same jurisdiction-urbanicity-race/ethnicity stratum combination.

Methodology

To determine whether the distributions of schools by achievement measure between the frame and school sample were different, comparisons of percentile estimates were made for the 10th, 25th, 50th, 75th, and 90th percentile levels as well as the mean for each public school jurisdiction by grade. Frame and school sample estimates were considered statistically different if the frame value fell outside the 95 percent confidence interval of the corresponding sample estimate. The percentile values for the frames were calculated by weighting each school by the estimated number of students in the given grade. The percentile estimates for the school samples were calculated using school weights and weighted by the school measure of size (estimated number of students in the given grade). The 95 percent confidence intervals for the school sample estimates were calculated in WesVar—software for computing estimates of sampling variance from complex sample survey (Westat, 2000b)—using the Woodruff method (Sarndal, Swensson, and Wretman 1992) with the use of a finite population correction factor.

Results

As mentioned above, sample and frame distributions of schools by achievement measure were determined to be different if at least one of the percentile estimates or the mean differed significantly at the 95 percent confidence level. Out of all the jurisdiction and grade comparisons (excluding jurisdictions where all schools in the frame were selected), only 64 of the 876 distributions compared were found to be significantly different. They are shown in the table below.

The number of significant differences found in this analysis was close to what would be expected, albeit slightly higher, given the large number of comparisons that were made. Also, the number of significant results were widely spread throughout different grades and jurisdictions. Even in the statistically significant cases, the close adherence of sample values to frame values suggests there is little evidence that the school sample for NAEP 2022 is not representative of the frame from which it was selected. The achievement/median-income variable is used as the third-level sort order variable in the school systematic selection procedure. While it may be a rather low level sort variable, it still helps control how representative the sampled schools are in terms of achievement. The close agreement between frame and sample values of these achievement/median-income variables provided assurance that the selected sample is representative of the frame with respect to achievement or income status.

Printer-friendly Version