- Surveys & Programs
- Data & Tools
- Fast Facts
- News & Events
- Publications & Products
- About Us
Table of Contents | Search Technical Documentation | References
NAEP Technical DocumentationCharacteristics of Schools Related to Response
 |
||
|
Results of the Logistic Regression Analysis of School Nonresponse |
To evaluate the possibility that substantial bias remains as a result of school nonparticipation following the use of nonresponse adjustments, a series of analyses were conducted on the response status for schools participating in 2000 state assessment. These analyses were restricted to those states with a participation rate below 90 percent (after substitution), because these are the states where the potential for nonresponse bias was likely to be the greatest. States with an initial school response rate below 70 percent were not included, since NAEP does not report results for these states because of concern about nonresponse bias. The table below gives a list of states that were included in the analysis for each grade and subject. The school participation rates (after substitution) are shown in parentheses.
| Grade | Subject | States included in logistic regression analysis |
|---|---|---|
| 4 | Mathematics | AZ (88%) AR (87%) CA(76%) ID (75%) IL(74%) IN (71%) IA(70%) KS(71%) ME(86%) MI(85%) MN(83%) MT(77%) NY(71%) ND(88%) OH(82%) OR(74%) VT(70%) |
| Science | AZ (87%) AR (85%) CA(76%) ID (75%) IL(73%) IN (70%) IA(71%) ME(86%) MI(84%) MN(83%) MT(77%) NY(72%) ND(89%) OH(82%) OR(74%) VT(75%) | |
| 8 | Mathematics | AZ (76%) AR (87%) CA(72%) ID (78%) IL(75%) IN (73%) KS(71%) ME(84%) MI(81%) MN(74%) MT(73%) OR(75%) VT(82%) |
| Science | AZ (76%) AR (87%) CA(72%) ID (78%) IL(75%) IN (73%) ME(85%) MI(81%) MN(73%) MT(74%) NY(71%) OR(74%) VT(80%) |
|
| SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000. | ||
The approach used was to develop a logistic regression model to predict the probability of participation as a function of the nonresponse adjustment classes and other school characteristics. These models were developed for schools in each of the states in the four grade-subject combinations specified in the above table. This resulted in the development of 59 models, which generally differ only in the number of nonresponse class levels that are included in the model. The number of final nonresponse adjustment classes varied by state. The logistic regression analysis was used to determine whether the response rates are significantly related to school characteristics, after accounting for the effect of the nonresponse class. Thus "dummy" variables were created to indicate nonresponse class membership.
If there are k nonresponse classes within a state, for i = 1, ... , (k - 1), let Xij = 1 if the school j is classified in nonresponse class i = 0 otherwise.
Within each state a logistic model was fit to the data on school participation. In the model, the indicator variables for the nonresponse class and additional variables available for participating and nonparticipating schools alike were included. These variables were the percentage of Black students (Y1), the percentage of Hispanic students (Y2), the estimated enrollment for grades 4 and 8 of the school (Y3), and a variable that indicates either achievement or the median household income of the ZIP Code area in which the school was located (Y4). Achievement data was used if available, otherwise median household income was used.
Let Pj denote the probability that school j is a participant, and let Lj denote the logit of Pj. That is,
.The model fitted in each state was as follows:
Note that this model cannot be estimated if there are nonresponse classes in which all schools participated (so that no adjustments for nonresponse were made for those schools). Even though this analysis was restricted to those states with lower response, inestimable cases occurred in a number of instances. When this happened, those (responding) schools in such classes were dropped from the analyses.
The linked tables at the top of this web page show the proportion of the student population for each state that is represented by schools that have less than 100 percent response in every nonresponse class.
In grade 4 mathematics for Indiana, New York, Ohio, and Vermont, there was some nonresponse with every nonresponse class (i.e., adjustment cell), evidenced in column 3 where one can see that 100% of the population is covered by the model. When the percent of the population covered by the model is less than 100%, some portion of the population is not represented because schools were dropped from nonresponse classes with no nonresponse. For grade 4 mathematics, for the states Arizona, Arkansas, California, Idaho, Illinois, Iowa, Kansas, Maine, Michigan, Minnesota, Montana, North Dakota, and Oregon, some portion of the student population is not represented because schools were dropped from classes with no nonresponse.
These tables show that only eight of the 59 models that contained all of the variables were significant at the p = 0.05 level. These were the models for Idaho, grade 4 mathematics and science, grade 8 mathematics for Idaho, Kansas, Montana, Vermont, and grade 8 science for Arkansas and Idaho, all with p-values ranging from 0.000 to 0.030. For some states, the overall model was not significant, but individual variables were significant (as noted in the table). Examples of such states are California, grade 4 mathematics, where the percent of Hispanic students was significant. In Idaho, grade 4 mathematics, the estimated grade enrollment was significant. Other examples where the overall model was not significant but contained significant individual variables were Iowa and Oregon grade 4 science, where percent Black and percent Hispanic, respectively, were significant. Illinois and Indiana, grade 8 science, were both significant for the percent of Hispanic students. Arkansas, grade 8 science, was significant for a particular nonresponse class and the estimated grade enrollment. Idaho, grades 4 and 8 science, and grade 8 mathematics for Arkansas, Idaho, and Montana were significant for particular nonresponse classes.
To determine if the variables other than the nonresponse adjustment cell variables added explanatory power to the model, all variables except the nonresponse adjustment cell variables were tested collectively to see if the estimates of the parameters were equal to zero. This evaluates whether, taken as a group, the Y variables are significantly related to the response probability, after accounting for nonresponse cell.
The results are shown in the last columns of these tables. Only two of the 59 tests were significant (grade 4 mathematics in Idaho and grade 8 science in Maine). The rest of the tests were not significant, which suggests that the variables did not add to the model after accounting for the nonresponse adjustment classes, even though, on occasion, an individual variable was significant. The score test, which also tests the overall fit of the model, was included in the software runs as well. Footnotes in the table indicate when the p-value for the score statistic was significant. This happened three times: grade 4 mathematics and science in California and grade 8 mathematics in Illinois. The results of the analysis indicate that on occasion there were differences between the originally sampled schools and those that participated, which were not fully removed by the process of creating nonresponse adjustments. Although these effects were not dramatic, they were sometimes statistically significant, and in these instances, this was reflected in noticeable differences in population characteristics between respondents and the full sample. However, the evidence presented here does not permit valid speculation about the likely size or even direction of the bias in achievement results in mathematics and science for the few states where these sample differences are noticeable.
Last updated 17 April 2008 (GF)
