Skip Navigation
small NCES header image

2009 Spotlight

U.S. Performance Across International Assessments of Student Achievement

Technical Notes: A.11 Nonresponse bias analysis

In accord with NCES statistical standard 4-4-1 (Seastrom 2003), nonresponse bias analyses are conducted for any survey stage of data collection with a unit or item response rate less than 85 percent. Estimates of survey characteristics for nonrespondents and respondents are compared on sampling frame variables, for which data on respondents and nonrespondents are available, so as to assess the potential nonresponse bias. Comparisons are made before and after weight adjustments for nonresponse bias, and these comparisons examine both bias and relative bias. Bias is calculated as the difference between the value of a survey characteristic for all schools (or students) that were sampled and the value of that characteristic for just those schools (or students) that actually responded. Relative bias reports this difference as a percentage of the value of the survey characteristic for all schools (or students).

For example, in PIRLS 2006, of the 214 U.S. schools in the original sample eligible to participate in PIRLS, 120 agreed to participate (for a response rate of 57 percent, weighted). An additional 63 substitute schools agreed to participate, bringing the total up to 183 schools for a weighted response rate, using final adjusted weights, of 86 percent of eligible schools (NCES 2009-050). With these substitute schools, the United States met the international guidelines for school response rates. However, a nonresponse bias analysis was conducted to determine what amount of bias and relative bias existed in the original and final sample. Such an analysis is considered to have found potential bias in a survey characteristic if either (a) a test of statistical significance indicates a significant difference between the responding and nonresponding sample percentages for that survey characteristics or (b) the relative bias for that survey characteristic is greater than 10 percent.

The PIRLS 2006 nonresponse bias analysis identified four variables as either (a) significant in bivariate or multivariate bias analysis or (b) having a relative bias of greater than 10 percent. The following list summarizes these variables and the difference found, after weight adjustments, between the value of the variable in the final sample of responding schools (which includes substitute schools) and value in the full sample from which the schools were drawn:

  • private schools (19.4 percent of the schools in the final sample vs. 17.0 percent of the schools in the full sample)
  • schools in the Northeast Census region (20.7 percent of schools in the final sample vs. 21.7 percent of schools in the full sample)
  • percentage of Asian or Pacific Islander students (2.6 percent of the students in schools in the final sample vs. 2.8 percent of the students in schools in the full sample)
  • percentage of White, non-Hispanic students (53.9 percent of the students in schools in the final sample vs. 53.5 percent of the students in schools in the full sample)

In TIMSS 2007, of the 290 U.S. schools in the original sample eligible to participate in TIMSS at 4th grade (TIMSS-4) and 287 at 8th grade (TIMSS-8), 202 schools at grade 4 and 197 schools at grade 8 agreed to participate (or, respectively, 70 and 68 percent, weighted). An additional 55 substitute schools for TIMSS-4 and 42 substitute schools for TIMSS-8 agreed to participate, bringing the total up to 257 schools at grade 4 and 239 schools at grade 8 for weighted response rates, using final adjusted weights, of 89 and 83 percent, respectively, of eligible schools (NCES 2009-xxx). With these substitute schools, the United States met the international guidelines for school response rates. However, a nonresponse bias analysis was conducted to determine what amount of bias and relative bias existed in the original and final sample.

The TIMSS 2007 nonresponse bias analysis identified several variables as either (a) significant in bivariate or multivariate bias analysis or (b) having a relative bias of greater than 10 percent. The following list summarizes these variables (in the 4th and 8th grade sample) and the difference found, after weight adjustments, between the value of the variable in the final sample of responding schools (which includes substitute schools) and value in the full sample from which the schools were drawn:

TIMSS-4

  • central city schools (29.6 percent of the schools in the final sample vs. 32.1 percent of the schools in the full sample)
  • rural schools (28.4 percent of the schools in the final sample vs. 26.2 percent of the schools in the full sample)
  • percentage of American Indian/Alaska Native students (1.6 percent of the students in the final sample vs. 1.4 percent of the students in schools in the full sample)
  • percentage of White, non-Hispanic students (60.9 percent of the students in schools in the final sample vs. 59.2 percent of the students in schools in the full sample)

TIMSS-8

  • central city schools (33.5 percent of the schools in the final sample vs. 30.0 percent of the schools in the full sample)
  • percentage of American Indian/Alaska Native students (1.5 percent of the students in the final sample vs. 1.3 percent of the students in schools in the full sample)
  • percentage of Black, non-Hispanic students (15.1 percent of the students in the final sample vs. 15.7 percent of the students in schools in the full sample)

In PISA 2006, of the 209 U.S. schools in the original sample eligible to participate in PISA, 145 agreed to participate (69 percent, weighted). An additional 21 substitute schools agreed to participate, bringing the total up to 166 schools for a 79 percent overall response rate (NCES 2009-055). With these substitute schools, the United States met the PISA standards for a minimum participation rate (65 percent of original sample schools). However, since the U.S. response rate level did not meet the NCES standard of 85 percent, the nonresponse bias analysis was conducted to shed light on the quality of the data.

The PISA 2006 nonresponse bias analysis identified eight variables as either (a) significant in bivariate or multivariate bias analysis or (b) having a relative bias of greater than 10 percent. The following list summarizes these variables and the difference found, after weight adjustments, between the value of the variable in the final sample of responding schools (which includes substitute schools) and the value in the full sample from which the schools were drawn:

  • central city schools (25.8 percent of the schools in the final sample vs. 22.8 percent of the schools in the full sample)
  • urban fringe or large town schools (24.0 percent of the schools in the final sample vs. 24.7 percent of the schools in the full sample)
  • schools in the South Census region (35.9 percent of schools in the final sample vs. 34.0 percent of schools in the full sample)
  • schools in the West Census region (15.6 percent of schools in the final sample vs. 20.4 percent of schools in the full sample)
  • percentage of American Indian/Alaska Native students (1.4 percent of the students in the final sample vs. 2.2 percent of the students in schools in the full sample)
  • percentage of Hispanic students (12.0 percent of the students in schools in the final sample vs. 10.8 percent of the students in schools in the full sample)
  • percentage of White, non-Hispanic students (61.5 percent of the students in schools in the final sample vs. 63.5 percent of the students in schools in the full sample)
  • percentage of students of Other race/ethnicity (2.9 percent of the students in the final sample vs. 2.5 percent of the students in schools in the full sample)
Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education