NAEP Technical DocumentationEvaluation of State Achievement Data in the Sampling Frame

The purpose of this analysis was to determine whether public schools selected for the NAEP 2005 samples were representative of the schools on the NAEP sampling frames in terms of achievement. Percentiles of the achievement distributions were compared between the frame and sample schools for each public school jurisdiction in grades 4 and 8.

Achievement Data

The achievement variable used in the analysis was the same variable used in NAEP sample design to stratify the public school frame. For most jurisdictions, the variable was an achievement score provided by the jurisdiction. However, for some jurisdictions where achievement data were not available, median household income from the 2000 Census was used. (In 2000, the Census determined median household income based on the five-digit ZIP Code area in which the school was located.) The achievement data consisted of various types of school-specific achievement measures from state assessment programs. The type of achievement data available varied by jurisdiction. For instance, in some states, the measure was the average score for a given state assessment. In other states, the measure was a percentile rank or percent of students above a specific score.

During frame development, not every record on the Common Core of Data (CCD) file matched to the achievement data files created for the National Center for Education Statistics (NCES), even in jurisdictions where those data were generally available. For schools that did not match, an attempt was made to impute an achievement score using a hotdeck imputation approach, but in some cases an adequate donor could not be found. Schools which persisted in having missing achievement values even after imputation were removed from the NAEP frame and sample data sets used in this analysis.

Methodology

To determine whether the distributions between the frame and sample schools were different, comparisons of quantile estimates were made for the 10th, 25th, 50th, 75th, and 90th percentile levels, as well as the mean, for each public school jurisdiction by grade. Frame and sample school estimates were considered statistically different if the frame value fell outside the 95 percent confidence interval of the corresponding sample estimate. The percentile values for the frame schools were calculated by weighting each school by the estimated number of students in the given grade. The percentile estimates for the sample schools were calculated using school weights and weighted by the school measure of size (estimated number of students in the given grade). The 95 percent confidence intervals for the school sample estimates were calculated in WesVar—software for computing estimates of sampling variance from complex sample survey (Westat 2000b)—using the Woodruff method (Sarndal, Swensson, and Wretman 1992 ) and without the use of a finite population correction factor. A finite population correction is not traditionally used in computing variances for NAEP estimates.

Results

Sample and frame achievement distributions were determined to be different if at least one of the percentile estimates or the mean differed significantly at the 95 percent confidence level. Out of all the jurisdiction and grade comparisons, only seven distributions were found to be significantly different. They are described below.

For grade 4 in Michigan, the difference between the frame value and sample estimates for the mean achievement score was statistically significant. The frame estimate was 64.86 compared to the sample estimate of 66.23, with a 95-percent confidence interval of (64.88, 67.59). Although the difference is significant, the frame estimate falls only 0.02 percentage point below the lower confidence limit for the sample. For grade 4 Puerto Rico, the difference between the frame and sample estimates at the 50th percentile level was statistically significant. The frame estimate was 45.36, whereas the sample estimate was 45.05, with a 95-percent confidence interval of (44.85, 45.25). Here the absolute difference is quite small (0.31 percentage point) and the frame value falls just above the upper confidence limit. For the Austin, Texas TUDA, the distributions significantly differed at the 75th percentile and the mean. For both statistics, the frame value is only a fraction of a percentage point above the upper confidence limit of the sample estimate (0.23% for the mean and 0.07% for the 75th percentile).

In grade 8, the frame and sample values for the 50th percentile in New York are significantly different. The frame estimate is 718.93, compared to a sample estimate of 718.07 (95% confidence interval: 716.58, 718.85). Once again the absolute difference in achievement scores is small (0.1 percent) and the frame value falls only 0.11 above the upper confidence limit of the sample estimate. In grade 8 Puerto Rico, the frame and sample estimates for the 10th percentile are significantly different. The frame estimate is 11,003.46, while the sample estimate is 10,994.25 (95% confidence interval: 10,991.51, 10,996.98). In this case as well, the absolute difference is small (less than 0.1 percent) and the frame value lies barely outside the confidence interval. Further, median income was used instead of achievement data in Puerto Rico for the eighth-grade sample, and while it is known that median income is a good proxy for achievement, the two are not perfectly correlated. For the Los Angeles, TUDA, the distributions differed at the 75th percentile. The frame value of the scale score fell 0.03 points below the lower confidence limit on the sample estimate. For the San Diego TUDA, the distributions also differed at the 75th percentile. Here the frame value of the scale score is 1.45 points higher than the upper confidence limit on the sample estimate—a difference of 0.2 percent.

The number of significant differences found in this analysis is smaller than what would be expected to occur by chance, given the large number of comparisons that were made. The small number of significant differences may be partially accounted for by the lack of use of a finite population correction factor in the calculation of the sampling variances. However, the close adherence of sample values to frame values suggests that there is little evidence that the school sample for NAEP 2005 is not representative of the frame from which it was selected. The achievement/median income variable is used as the fourth-level sort order variable in the school systematic selection procedure. While it may be a rather low-level sort variable, it still helps control how representative the sampled schools are in terms of achievement. The close agreement between frame and sample values of these achievement/median income variables provides assurance that the selected sample is representative of the frame with respect to achievement status.

Additional detail can be found in the report Supplemental Tables from NAEP 2005 Sample Design (Westat 2005).

Last updated 16 April 2009 (RF)

Printer-friendly Version