- Surveys & Programs
- Data & Tools
- Fast Facts
- News & Events
- Publications & Products
- About Us
Table of Contents | Search Technical Documentation | References
NAEP Technical DocumentationCharacteristics of Nonresponding Schools and Students
Survey weights are adjusted so as to reduce the potential bias of nonparticipation of sampled schools and students. To the extent that the characteristics of nonresponding schools or students are different from those of respondents in the same nonresponse adjustment class, potential for nonresponse bias remains.
The potential for remaining nonresponse bias was present in 2000 state assessment in two related ways and is examined in the following section. First, weighted distributions for each grade and subject within each jurisdiction, of certain characteristics of schools and students, both for the full sample and for respondents only, are discussed. This analysis is of necessity limited to those characteristics that are known for both respondents and nonrespondents, and hence, cannot directly address the question of nonresponse bias. The approach taken does reflect the reduction in bias obtained through the use of nonresponse weighting adjustments. As such, it is more appropriate than a simple comparison of the characteristics of nonrespondents with those of respondents for each subject and jurisdiction.
The second approach involves modeling the probability that a school is a respondent, as a function of the nonresponse adjustment class to which the school belongs, together with other school characteristics. This was achieved using linear logistic regression models, with school response status as the dependent variable. By testing to see if the school characteristics add any predictive ability to the model over using the membership of the nonresponse adjustment class to make this prediction, we can obtain some insight into the remaining potential for nonresponse bias. If these factors are substantially marginally predictive, there is a danger that significant nonresponse bias remains.
To study the potential for nonresponse bias, analysts compared the school characteristics of responding schools with those of the full sample. The means that are presented in the tables that follow are the percentage of Black students in the school, the percentage of Hispanic students, the median household income (1989) of the ZIP Code area where the school is located, mean achievement and the type of location. The first two variables were obtained from the sample frame, and hence from the Common Core of Data (CCD). Median income was obtained from the 1990 Donnelly File. The achievement data was only available from certain jurisdictions. The variable designating type of location was provided by CCD. The type of location variable has seven possible levels. Although this variable is not interval-scaled, the mean value does give an indication of the degree of urbanization of the population represented by the school sample (lower values for type of location indicate a greater degree of urbanization).
The mean values of the variables, both for the responding schools and the full sample, were calculated for all jurisdictions.
The means are weighted appropriately to reflect whether nonresponse adjustments have been applied (i.e., to respondents only) or not (to the full set of eligible schools). For each grade and subject, two sets of means are presented for these variables. The first set shows the weighted mean derived from the full sample of in-scope schools selected for each subject; that is, respondents and nonrespondents (for which there was no participating substitute). The weight for each sampled school is the product of the school base weight and the grade enrollment. This weight therefore represents the number of students in the jurisdiction represented by the selected school. The second set of means is derived from responding schools only, after school substitution. In this case the weight for each school is the product of the nonresponse-adjusted school weight and the grade enrollment of the original school, and therefore indicates the number of students in the jurisdiction represented by the responding school.
The differences between these sets of means give an indication of the potential for nonresponse bias that has been introduced by nonresponding schools with no participating substitute. For example, for grade 4 mathematics in California, the mean percentage Hispanic enrollment, estimated from the original sample of schools, is 43 percent. The estimate from the responding schools is 39 percent. Thus there may be a slight bias in the results for California because these two means differ. Note, however, that throughout the four tables referred to here, the differences in the two sets of mean values are generally very slight, at least in absolute terms, suggesting that it is unlikely that substantial bias has been introduced by schools that did not participate and for which no substitute participated. In a number of states, there was no nonresponse at the school level (weighted participation rate is 100 percent), so that these sets of means are identical. Even in those jurisdictions where school nonresponse was relatively high (such as in Wisconsin grade 4 mathematics and science), the absolute differences in means are slight. Occasionally the relative difference is larger; for instance, the weighted mean value for "Percent Black" in New York for grade 8 science is 19 derived from the full sample and 22 derived from the responding schools. However, these are for small population subgroups, and thus are very unlikely to have a large impact on results for the jurisdiction as a whole. For additional information concerning these characteristics, refer to the following sections:
Last updated 13 August 2008 (KL)
