The statistics in this report are estimates derived from a sample. Two broad categories of error occur in such estimates: sampling and nonsampling errors. Sampling errors occur because observations are made on only samples of students, not entire populations. Nonsampling errors occur not only in sample surveys but also in complete censuses of entire populations. Nonsampling errors can be attributed to a number of sources: inability to obtain complete information about all students in all institutions in the sample (some students or institutions refused to participate, or students participated but answered only certain items); ambiguous definitions; differences in interpreting questions; inability or unwillingness to give correct information; mistakes in recording or coding data; and other errors of collecting, processing, sampling, and imputing missing data. Readers interested in efforts to minimize nonsampling errors for estimates used in this report should consult the methodology reports referenced earlier in this appendix. Below is a discussion on possible bias on statistics for a couple of variables presented in the tables/figures of this report that had low item response rates.
Weighted item response rates were calculated for all the variables used in this report by dividing the weighted number of valid responses by the weighted population for which the item was applicable. Overall, most of the items had very high response rates.
Only two variables had weighted item response rates below 85 percent. In one of these cases (B3RPYTYP, type of loan repayment in 2003), the low weighted response rate18 percentis due largely to the fact that this variable was applicable to a small proportion of the sample population (i.e., those who were in repayment in 2003), hence leaving a large proportion of the sample population with incomplete interviews. Such cases are considered to have indeterminate responses, as are respondents who give invalid responses (such as “Refused” or “Don’t know”). Incomplete interviews thus make up a relatively high proportion of the indeterminate responses for this item. However, it is highly likely that the majority of indeterminate responses would have been excluded from the item had their information been gathered, considering that the item applies only to a small proportion of the sample population. When incomplete interviews were excluded from the calculation of the item response rate, the response rate for B3RPYTYP indeed increased from 18 to 81 percent, a big improvement but nonetheless still below the NCES threshold of 85 percent. However, the only incidence where this variable was used in this report is when the focus is on those who had no additional degree enrollment and were in repayment in 2003 (table 8), a subgroup of the sample population, for which the item response rate for B3RPYTYP is actually 99 percent. Thus, it is very unlikely that statistics presented in table 8 and relevant statements made in the text are biased because of missing data.
The only other variable with a weighted item response rate below 85 percent is B2SALARY (annual salary at April 1997 job) when it was used to compute the median and average salaries in 1997 in table 1 for the entire applicable sample (rather than a subgroup of the sample). A bias analysis was conducted to determine whether the cases missing values for this variable differed from those with positive values in aspects that are associated with salary income. Cases with missing and positive responses were compared with each other for four demographic variables: GENDER (gender), RETHNIC (race/ethnicity), BAMAJOR (undergraduate major), and SECTOR_B (degree-granting institution type). Each of these comparison variables had a response rate of 96 percent or higher and was related to B2SALARY.
Results show that there were no measurable gender-associated differences between respondents who had positive values on B2SALARY and those with missing values for this variablee.g., the percentage of males was 45 and 44 percent, respectively. However, those with missing values for B2SALARY were more likely than those with valid data to have been Asian/Pacific Islander (8 vs. 4 percent) but less likely to have been White (80 vs. 84 percent) and more likely to have graduated from private not-for-profit doctoral institutions (17 vs. 13 percent), characteristics associated with higher salary income ($37,500 for Asian/Pacific Islander vs. $32,500 for White; $36,400 for private not-for-profit doctoral institutions vs. $30,400$33,100 for others). This suggests the possibility that the statistics reported in the table might have been underestimatedthat is, the average and median salary would likely have been higher if the response rate for B2SALARY had been higher. However, respondents with unknown values for B2SALARY were more likely than those with known values to have majored in humanities and social sciences (27 vs. 23 percent) and less likely to have majored in the “Other” category (22 vs. 25 percent), which would likely lead to estimates lower than those presented in table 1, because humanities and social sciences majors earned, on average, less than those whose major was in the “Other” category ($29,700 vs. $33,400). Nonetheless, in neither direction of potential bias were the differences between respondents and nonrespondents considerable in magnitude, meaning that if there were any biases, they would have had a very limited effect on the overall sample. When combining this with the fact that among all sample cases, only 19 percent of them had a missing value on B2SALARY, it is unlikely that the estimates reported in table 1 would be seriously biased.