NAEP Technical DocumentationMain Quality Control Findings of Interest: Nonresponse Bias Analyses for the 2007 Assessment

School-level nonresponse bias analyses were conducted on private schools at all three grades (4, 8, and 12) since the school-level participation rate of each of these reporting groups fell below the 85 percent participation threshold. The school analysis was conducted in three parts. The first part of the analysis looked for the potential for nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part examined the remaining potential for bias after accounting for the mitigating effects of both substitution and nonresponse weight adjustments.

Overall, it was found that the school nonresponse adjustments reduced the potential for nonresponse bias among several variables. For grades 4 and 8 private schools, substitution and school nonresponse adjustments appear to have been at least partially effective in reducing the nonresponse bias, as there are no significant indications of bias for these samples after substitution and nonresponse adjustment among over 20 domains analyzed. For grade 12 private schools, a significant race/ethnicity result occurred for the percent Hispanic estimate. The absolute bias is small (1.5 percent on estimates of about 8 percent), but the relative bias is large (18.3 percent). For a bias this size, and for a population this size, there is potential for a very minimal effect on the test scores on the overall sample. Still, there should be some caution when analyzing Hispanic student data for private schools.

Student-level nonresponse bias analyses were conducted on grade 12 public schools since the student-level participation rates of these reporting groups fell below the 85 percent participation threshold. The analyses were conducted in two parts. The first part of the analysis indicated the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments.

Among over 15 domains analyzed, a statistically significant difference resulted for the percentage of students who are English language learners (ELL). The significant ELL result is associated with a small absolute bias (0.2 percent on estimates of about 3.4 percent), and a moderate level of relative bias (5.2 percent). Bias at this level has a potential for a very minimal effect on estimates.

To see the entire report or detailed tables, see NAEP 2007 Nonresponse Bias Analysis Report. (PDF 189KB)

Last updated 28 April 2010 (GF)

Printer-friendly Version