U.S. Nonresponse Bias Analysis

Return to Methodology and Technical Notes

Of the 240 original sampled schools in the United States national sample, 213 were eligible (18 schools did not have any 15-year-olds enrolled, 6 had closed, and 3 were otherwise ineligible), and 142 agreed to participate. Of the 58 original sampled schools in the Massachusetts state sample, 53 were eligible (5 schools did not have any 15-year-olds enrolled), and 41 agreed to participate. The weighted school response rate before replacement was 67 and 77 percent for the United States and Massachusetts, respectively, requiring the United States to conduct a nonresponse bias analysis, which was used by the PISA consortium and the OECD to evaluate the quality of the final United States sample.

A bias analysis was conducted in the United States to address potential problems in the data owing to school nonresponse. The general approach taken involves an analysis in three parts as described below.

1. Analysis of the participating original sample: The distribution of the participating original school sample was compared with that of the total eligible original school sample. The participating original sample is the sample before substitution. In each sample, schools were weighted by their school base weights and enrollment of eligible students, excluding any nonresponse adjustment factor.
2. Analysis of the participating final school sample with substitutes: The distribution of the participating final school sample, which includes participating substitutes that were used as replacements for nonresponding schools from the eligible original sample, was compared to the total eligible final school sample. The total eligible final sample includes the participating final sample plus those original nonrespondents that were not replaced by substitutes. Again, schools were weighted by their school base weights and enrollment of eligible students for both the eligible sample and the participating schools.
3. Analysis of the nonresponse adjusted final sample with substitutes: The same sets of schools were compared as in the second analysis, but this time, when analyzing the participating final schools alone, school nonresponse adjustments were applied to the weights.

To compare PISA participating schools to the total eligible sample of schools, it was necessary to match the sample of schools to the sample frame to identify as many characteristics as possible that might provide information about the presence of nonresponse bias. Frame characteristics were taken from the 2012�13 Common Core of Data for public schools and from the 2011�12 Private School Universe Survey for private schools. The available school characteristics included affiliation (public or private), locale (central city, suburb, town, rural), Census region, number of age-eligible students, total number of students, and percentage of various racial/ethnic groups (White, non-Hispanic; Black, non-Hispanic; Hispanic; Asian; American Indian or Alaska Native; Hawaiian/Pacific Islander; and two or more races). The percentage of students eligible for free or reduced-price lunch was available for public schools only. Since the Massachusetts state sample only includes public and not private schools, school control in addition to census region do not apply and are not included in any of the analyses. Additionally, only a regression model with public schools could be conducted. The full text of the nonresponse bias analysis conducted for PISA 2015 will be included in the technical report released with the U.S. national dataset (Kastberg, Roey, Lemanski, Murray, and Ferraro forthcoming).

For the United States original sample schools, schools in the Northeast were underrepresented among participating schools relative to eligible schools (12.6 vs. 17.1 percent, respectively), while schools in the South were overrepresented among participating schools (43.3 vs. 37.8 percent, respectively). Participating schools had a lower mean percentage of White, non-Hispanic students than the eligible sample (49.1 vs. 53.1 percent, respectively) and a higher mean percentage of Hispanic students than the eligible sample (27.4 vs. 24.6 percent, respectively). Additionally, the absolute value of the relative bias for private schools and schools in towns is greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. When all factors were considered simultaneously in a logistic regression analysis, none of the parameter estimates were significant predictors of participation. The percentage of students eligible for free or reduced-price lunch was not included in the logistic regression analysis as public and private schools were modeled together using only the variables available for all schools.

For the United States final sample schools (with substitutes), there were no statistically significant relationships between participation status and any of the characteristics studied. However, the absolute value of the relative bias for private schools, schools in towns and the Northeast region are greater than 10 percent, which indicates potential bias even though no statistically significant relationships were detected. When all factors were considered simultaneously in a logistic regression analysis (again with free or reduced-price lunch eligibility omitted), no variables were statistically significant predictors of participation.

In the United States final sample schools with substitutes when school nonresponse adjusted weights were used for the participating schools, there were no statistically significant relationships between participation status and any of the characteristics studied. We therefore conclude that there is little evidence of resulting potential bias in the final sample. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse adjusted weights.

In sum, the investigation into nonresponse bias at the school level in the United States in PISA 2015 provides evidence that there is little potential for nonresponse bias in the PISA participating sample based on the characteristics studied. It also suggests that the use of substitute schools substantially reduced the potential for bias. Moreover, after the application of school nonresponse adjustments, there is no evidence of resulting potential bias in the final sample.

For the Massachusetts original sample schools, no variables were found to be statistically significantly related to participation in the bivariate analysis. However, the absolute value of the relative bias for schools in towns and rural areas are greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. When all of these factors were considered simultaneously in a regression analysis (with public schools only), only town is a significant predictor of participation among public schools only.

For the Massachusetts final sample of schools (with substitutes), schools in the central cities were underrepresented among participating schools relative to eligible schools (12.3 vs. 13.9 percent, respectively), while schools in rural areas were overrepresented among participating schools (12.3 vs. 11.5 percent, respectively); participating schools had a higher mean percentage of White, non-Hispanic students than the eligible sample (71.8 vs. 70.3 percent, respectively) and a lower mean percentage of Black, non-Hispanic students than the eligible sample (7.4 vs. 8.7 percent, respectively). When all of these factors were considered simultaneously in a regression analysis (with public schools only), central city and suburb are significant predictors of school participation among public schools only.

In the Massachusetts final sample of schools with substitutes when school nonresponse-adjusted weights were used for the participating schools, participating schools had a higher mean percentage of White, non-Hispanic students than the eligible sample (71.3 vs. 70.3 percent, respectively) and a lower mean percentage of Black, non-Hispanic students than the eligible sample (7.5 vs. 8.7 percent, respectively). We therefore conclude that there is some evidence of resulting potential bias in the final sample. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse-adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse-adjusted weights.

In sum, the investigation into nonresponse bias at the school level in Massachusetts in PISA 2015 provides evidence that there is some potential for nonresponse bias in the PISA participating original sample based on the characteristics studied. It also suggests that the use of substitute schools added to the potential for bias. This is due to all eight substitute schools being from either suburbs (six) or rural areas (two) and none from central cities. Moreover, the application of school nonresponse adjustments reduced the bias but there is still evidence of resulting potential bias in the final sample.

Top