Return to Methodology and Technical Notes
Since the U.S. PISA weighted school response rates are below 85 percent, NCES requires an investigation into the potential magnitude of nonresponse bias at the school level in the U.S. sample. The investigation into nonresponse bias at the school level for the U.S. PISA effort shows statistically significant relationships between response status and some of the available school characteristics that were examined in the analyses.
The general approach taken involves an analysis in three parts as described below.
The first analysis indicates the potential for nonresponse bias that was introduced through school nonresponse. The second analysis suggests the remaining potential for nonresponse bias after the mitigating effects of substitution have been accounted for. The third analysis indicates the potential for bias after accounting for the mitigating effects of both substitution and nonresponse weight adjustments. Both the second and third analyses, however, may provide an overly optimistic scenario, resulting from the fact that substitution and nonresponse adjustments may correct somewhat for deficiencies in the characteristics examined, but there is no guarantee that they are equally as effective for other characteristics and, in particular, for student achievement.
In addition to these tests, logistic regression models were used to provide a multivariate analysis that examined the conditional independence of these school characteristics as predictors of participation. The logistic regression compared frame characteristics for participating schools with non-participating schools, which is effectively the same as comparing the participating schools to the eligible sample as in the bivariate analysis.
Multivariate analysis can provide additional insights, over and above those gained through the bivariate analysis. It may be the case that only one or two variables are actually related to participation status. However, if these variables are also related to the other variables examined in the analyses, then other variables, which are not related to participation status, will appear as significant in simple bivariate tables. Multivariate analysis, in contrast, examines the conditional relationships with participation after controlling for the other predictor variables—thereby, testing the robustness of the relationships between school characteristics and participation.
Participating PISA schools and the total eligible PISA school sample were compared by as many school sampling frame characteristics as possible that might provide information about the presence of nonresponse bias. Comparing frame characteristics between participating schools and the total eligible school sample is not an ideal measure of nonresponse bias if the characteristics are unrelated or weakly related to more substantive items in the survey; however, often it is the only approach available since PISA data are not available for nonparticipating schools. While the school-level characteristics used in these analyses are limited to those available in the sampling frame, each of the variables had a demonstrated relationship to achievement in previous PISA cycles.
A summary of the findings is provided below. Additional details on the nonresponse bias analysis can be found in NCES’ Technical Report and User Guide for the 2018 Program for International Student Assessment (PISA) (Kastberg et al. forthcoming).
For original sample schools (not including substitute schools), nine variables were found to be statistically significantly related to participation in the bivariate analysis: school control, census region, poverty level, total school and age-eligible enrollments, White, non-Hispanic, Black, non-Hispanic, Hispanic, and free or reduced-price lunch. Additionally, the absolute value of the relative bias for small sized and large sized schools, American Indian or Alaska Native, and Hawaiian/Pacific Islander is greater than 10 percent, which indicates potential bias even though no statistically significant relationship was detected. Although each of these findings indicates some potential for nonresponse bias, when all of the factors (with seven race/ethnicity variables) were considered simultaneously in a regression analysis, the Northeast region, high poverty, and Two or more races were significant predictors of school participation. The second model (with summed race/ethnicity percentage) showed that high poverty was a significant predictor of participation. When all of the parameter estimates (with seven race/ethnicity variables) were considered simultaneously in a regression analysis, the Northeast region, high poverty, and Two or more races were significant predictors of participation. The second model (with summed race/ethnicity percentage) showed that high poverty and the summed race/ethnicity percentage were significant predictors of participation. The third model (with summed race/ethnicity percentage using public schools only) showed that high poverty was a significant predictor of school participation among public schools only.
For the final sample of schools (with substitute schools) with school nonresponse adjustments applied to the weights, no variables were found to be statistically significantly related to participation in the bivariate analysis. However, the absolute value of the relative bias for small sized schools and Hawaiian/Pacific Islander is greater than 10 percent. The multivariate regression analysis cannot be conducted after the school nonresponse adjustments are applied to the weights. The concept of nonresponse-adjusted weights does not apply to the nonresponding units, and, thus, we cannot conduct an analysis that compares respondents with nonrespondents using nonresponse-adjusted weights.
In sum, the investigation into nonresponse bias at the school level in the U.S. PISA 2018 data provides evidence that there is some potential for nonresponse bias in the PISA participating original sample based on the characteristics studied. It also suggests that, while there is some evidence that the use of substitute schools reduced the potential for bias, it has not reduced it substantially. However, after the application of school nonresponse adjustments, there is little evidence of resulting potential bias in the available frame variables and correlated variables in the final sample.
8 The size-adjusted weight modifies the PPS weight so that schools with relatively small number of students (and large school base weights) won't influence the results more than schools with relatively large number of students (and small school base weights).