International and U.S. Response Rates - U.S. Nonresponse Bias Analysis

Return to Methodology and Technical Notes

Of the 240 original sampled schools in the U.S. national sample, 207 were eligible (18 schools did not have any 15-year-olds enrolled, 6 had closed, and 9 were otherwise ineligible), and 139 agreed to participate. The weighted school response rate before replacement was 67 percent, requiring the United States to conduct a nonresponse bias analysis, which was used by the PISA consortium and the Organization for Economic Cooperation and Development (OECD) to evaluate the quality of the final sample.

A bias analysis was conducted in the United States to address potential problems in the data owing to school nonresponse. To compare PISA participating schools to the total eligible sample of schools, it was necessary to match the sample of schools to the sample frame to identify as many characteristics as possible that might provide information about the presence of nonresponse bias. Frame characteristics were taken from the 2008–09 Common Core of Data for public schools and from the 2009–10 Private School Universe Survey for private schools. The available school characteristics included affiliation (public or private), locale (city, suburb, town, rural), Census region, number of age-eligible students, total number of students, and percentage of various racial/ethnic groups (White, Black, Hispanic, non-Hispanic, Asian, American Indian or Alaska Native, Native Hawaiian/Pacific Islander, and multiracial). The percentage of students eligible for free or reduced-price lunch was available for public schools only. The full text of the nonresponse bias analysis conducted for PISA 2012 will be included in the technical report released with the U.S. national dataset (Kastberg, Roey, Lemanski, Chan, and Murray forthcoming).

For original sample schools, participating schools had a higher mean percentage of Hispanic students than the total eligible sample of schools (21.1 versus 18.1 percent, respectively). Participating original sample schools also had a higher mean percentage of students eligible for free or reduced-price lunch than did the total eligible sample of schools (39.3 versus 36.1 percent, respectively). When all factors were considered simultaneously in a logistic regression analysis, only “town” (a territory inside an urban cluster with a core population between 25,000 and 50,000) was a significant predictor of participation. The percentage of students eligible for free or reduced-price lunch was not included in the logistic regression analysis as public and private schools were modeled together using only the variables available for all schools.¹

For final sample schools (with substitutes), participating schools had a higher mean percentage of students eligible for free or reduced-price lunch than the total eligible sample of schools (38.4 versus 36.2 percent, respectively). When all factors were considered simultaneously in a logistic regression analysis (again with free or reduced-price lunch eligibility omitted), no variables were statistically significant predictors of participation.

With the inclusion of substitute schools and school nonresponse adjustments applied to the weights, only the percentage of students eligible for free or reduced-price lunch remained statistically significant. Specifically, the participating schools had a higher mean percentage of students eligible to receive free or reduced-price lunch than the total eligible sample of schools (38.4 versus 36.2 percent, respectively). However, there was not a statistically significant relationship between participating schools and the total frame of eligible schools for the percentage of students eligible for free or reduced-price lunch (38.4 versus 37.1 percent, respectively). We therefore conclude that, despite the tendency of schools with higher percentages of students eligible for free and reduced-price lunch to participate at a greater rate than other sampled schools, there is little evidence of resulting potential bias in the final sample. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse adjusted weights.

In sum, the investigation into nonresponse bias at the school level in the United States in PISA 2012 provides evidence that there is little potential for nonresponse bias in the PISA participating sample based on the characteristics studied. It also suggests that, while there is little evidence that the use of substitute schools reduced the potential for bias, it has not added to it. Moreover, the application of school nonresponse adjustments substantially reduced the potential for bias.

---