U.S. Nonresponse Bias Analysis

Previous Return to Methodology and Technical Notes

Since the U.S. PISA weighted school response rates are below 85 percent, NCES requires an investigation into the potential magnitude of nonresponse bias at the school level in the U.S. sample. The investigation into nonresponse bias at the school level for the U.S. PISA effort shows statistically significant relationships between response status and some of the available school characteristics that were examined in the analyses.

The general approach taken involves an analysis in three parts as described below.

  1. Analysis of the participating original sample: The distribution of the participating original school sample was compared with that of the total eligible original school sample. The participating original sample is the sample before substitution. In each sample, schools were weighted by their school base weights and enrollment of age-eligible students, referred to as a size-adjusted weight8, excluding any nonresponse adjustment factor. The base weight for each original school is the reciprocal of its selection probability.
  2. Analysis of the participating final school sample with substitutes: The distribution of the participating final school sample, which includes participating substitutes that were used as replacements for nonresponding schools from the eligible original sample, was compared to the total eligible final school sample. The total eligible final sample includes the participating final sample plus those original nonrespondents that were not replaced by substitutes. Again, schools were weighted by their size-adjusted school base weights for both the eligible sample and the participating schools. The base weight for each substitute school is the reciprocal of its selection probability.
  3. Analysis of the nonresponse adjusted final sample with substitutes: The same sets of schools were compared as in the second analysis, but this time, when analyzing the participating final schools alone, school nonresponse adjustments were applied to the size-adjusted school base weights. The international weighting procedures form nonresponse adjustment classes by cross classifying the explicit and implicit stratification variables. The eligible sample were again weighted by their size-adjusted school base weights.

The first analysis indicates the potential for nonresponse bias that was introduced through school nonresponse. The second analysis suggests the remaining potential for nonresponse bias after the mitigating effects of substitution have been accounted for. The third analysis indicates the potential for bias after accounting for the mitigating effects of both substitution and nonresponse weight adjustments. Both the second and third analyses, however, may provide an overly optimistic scenario, resulting from the fact that substitution and nonresponse adjustments may correct somewhat for deficiencies in the characteristics examined, but there is no guarantee that they are equally as effective for other characteristics and, in particular, for student achievement.

In addition to these tests, logistic regression models were used to provide a multivariate analysis that examined the conditional independence of these school characteristics as predictors of participation. The logistic regression compared frame characteristics for participating schools with non-participating schools, which is effectively the same as comparing the participating schools to the eligible sample as in the bivariate analysis.

Multivariate analysis can provide additional insights, over and above those gained through the bivariate analysis. It may be the case that only one or two variables are actually related to participation status. However, if these variables are also related to the other variables examined in the analyses, then other variables, which are not related to participation status, will appear as significant in simple bivariate tables. Multivariate analysis, in contrast, examines the conditional relationships with participation after controlling for the other predictor variables—thereby, testing the robustness of the relationships between school characteristics and participation.

Participating PISA schools and the total eligible PISA school sample were compared by as many school sampling frame characteristics as possible that might provide information about the presence of nonresponse bias. Comparing frame characteristics between participating schools and the total eligible school sample is not an ideal measure of nonresponse bias if the characteristics are unrelated or weakly related to more substantive items in the survey; however, often it is the only approach available since PISA data are not available for nonparticipating schools. While the school-level characteristics used in these analyses are limited to those available in the sampling frame, each of the variables had a demonstrated relationship to achievement in previous PISA cycles.

A summary of the findings is provided below. Additional details on the nonresponse bias analysis can be found in NCES’ Technical Report and User Guide for the 2018 Program for International Student Assessment (PISA) (Kastberg et al. forthcoming).

For original sample schools (not including substitute schools), nine variables were found to be statistically significantly related to participation in the bivariate analysis: school control, census region, poverty level, total school and age-eligible enrollments, White, non-Hispanic, Black, non-Hispanic, Hispanic, and free or reduced-price lunch. Additionally, the absolute value of the relative bias for small sized and large sized schools, American Indian or Alaska Native, and Hawaiian/Pacific Islander is greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. Although each of these findings indicates some potential for nonresponse bias, when all of the factors (with seven race/ethnicity variables) were considered simultaneously in a regression analysis, the Northeast region, high poverty, and Two or more races were significant predictors of school participation. The second model (with summed race/ethnicity percentage) showed that high poverty was a significant predictor of participation. When all of the parameter estimates (with seven race/ethnicity variables) were considered simultaneously in a regression analysis, the Northeast region, high poverty, and Two or more races were significant predictors of participation. The second model (with summed race/ethnicity percentage) showed that high poverty and the summed race/ethnicity percentage were significant predictors of participation. The third model (with summed race/ethnicity percentage using public schools only) showed that high poverty was a significant predictor of school participation among public schools only.

For the final sample of schools (with substitute schools) with school nonresponse adjustments applied to the weights, no variables were found to be statistically significantly related to participation in the bivariate analysis. However, the absolute value of the relative bias for small sized schools and Hawaiian/Pacific Islander is greater than 10 percent. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse-adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse-adjusted weights.

In sum, the investigation into nonresponse bias at the school level in the U.S. PISA 2018 data provides evidence that there is some potential for nonresponse bias in the PISA participating original sample based on the characteristics studied. It also suggests that, while there is some evidence that the use of substitute schools reduced the potential for bias, it has not reduced it substantially. However, after the application of school nonresponse adjustments, there is little evidence of resulting potential bias in the available frame variables and correlated variables in the final sample.

8 The size-adjusted weight modifies the PPS weight so that schools with relatively small number of students (and large school base weights) won't influence the results more than schools with relatively large number of students (and small school base weights).