Skip Navigation
PEDAR: Research Methodology Debt Burden - A Comparison of 1992–93 and 1999–2000 Bachelor’s Degree Recipients a Year After Graduating
Baccalaureate and Beyond Longitudinal Study
Overall Response Rates
Weight Variables
Accuracy of Estimates
Item Response Rates and Bias Analysis
Data Analysis System
Statistical Procedures
Differences Between Means
Executive Summary
Full Report (PDF)
Executive Summary (PDF)
 Accuracy of Estimates - Item Response Rates and Bias Analysis

Weighted item response rates were calculated for all the variables used in this report by dividing the weighted number of valid responses by the weighted population for which the item was applicable. Overall, most of the items had very high response rates. Items with weighted item response rates below 90 percent are shown in table B-1. Five variables had weighted item response rates below 85 percent. In one of these cases (CECURJOB, the primary reason that the respondent did not consider his or her current job the start of a career), the low weighted response rate is due largely to the fact that the variable was applicable to a small proportion of the sample population, yet that proportion included a relatively high percentage of respondents with incomplete interviews. Such respondents are considered to have indeterminate responses, as are respondents who give invalid responses (such as “Refused” or “Don’t know”). Incomplete interviews thus make up a relatively high proportion of the indeterminate responses for this item. However, it is highly likely that the majority of incomplete interviews would have been excluded from the item had their information been gathered, considering that the item applies only to a small proportion of the sample population. When incomplete interviews were excluded from the calculation of the item response rate, the response rate for CECURJOB changed from 81.8 to 97.5 percent. Therefore, for this variable, it is unlikely that reported differences are biased because of missing data.

GPAMAJ (grade-point average in undergraduate major) from B&B:2000/01 is one of the remaining four variables that has an item response rate below 85 percent. Because this variable was used only once in this report and was presented as a row variable for table 5 on graduate school enrollment a year later, respondents who were missing on GPAMAJ could actually be examined directly and separately (as already done in the table). Respondents missing on GPAMAJ had a graduate school enrollment rate similar to the rate of respondents with a GPA less than 3.0 (17 vs. 15 percent) , both of which were lower than the rate for respondents with a GPA of 3.0 or higher (24 percent). However, the reverse was observed when considering the likelihood of attending graduate school in the future if respondents had not yet done so (67, 60, and 69 percent, respectively). Because respondents missing GPA information appear to differ from those with this information, “missing” is shown as a response category.

One of the three remaining variables with an item response rate below 85 percent, ALLOWER (monthly payment on undergraduate loans) from B&B:93/94, had a response rate of 84 percent. This variable was used as a column variable (table 11) and to compute EDPCTR. A bias analysis was conducted to determine whether the cases missing values for this variable differed from those with positive values. Cases with missing and positive responses were compared with each other for all the row variables in table 11: GENDER (gender), RETHNIC (race/ethnicity), BAMAJOR (undergraduate major), INCQUTIL (dependency and family income), SECTOR_B (degree-granting institution type), TOTDEBT (total undergraduate debt), and B2EN9404 (enrollment status in 1994). Each of these comparison variables had a response rate of 96.6 percent or higher.

Results show that compared with respondents who had positive values on ALLOWER, those with missing values for this variable were less likely to be dependent students from the lowest family income group (14 vs. 21 percent), more likely to have been dependent students in the two highest family income groups (15 and 11 percent vs. 9 and 6 percent), and more likely to have graduated from private not-for-profit doctoral institutions (17 vs. 13 percent). These characteristics were associated with higher monthly payments (table 11). This suggests the possibility that the average reported in the table might have been underestimated—that is, the average would have been higher if the response rate for ALLOWER had been higher. However, respondents with unknown values for ALLOWER were less likely than those with known values not to be enrolled (85 vs. 93 percent), but they were more likely to be enrolled full time (9 vs. 3 percent), which would likely lead to a lower monthly payment (table 11). Nonetheless, in neither possibility of potential bias were the differences between respondents and nonrespondents considerable in magnitude, meaning that if there were any biases, they would have had a limited effect on the overall sample. When combining this with the fact that among all applicable cases for ALLOWER, only 16 percent of them had a missing value, it is unlikely that the estimates reported in table 11 would be seriously biased.

For EDPCTR from B&B:2000/01, which measures debt burden (monthly payment as a percentage of monthly income), cases with missing and valid responses were compared with each other for all the row variables in table 14: GENDER (sex), MAJORS4 (undergraduate major), SECTOR9 (degree-granting institution type), TOTDEBT (total undergraduate debt), EMPOLF (employment status in 2001), and CEANNERN (income quarters of salary in 2001).

Results show that compared with respondents who had valid values on EDPCTR, those with missing values for this variable were more likely to be male (47 vs. 43 percent) and to have majored in business and management (28 vs. 20 percent) and conversely, they were less likely to be female (53 vs. 57 percent), to have majored in education or humanities majors (7 vs. 9 percent and 23 vs. 29 percent, respectively). Since males had lower median debt burden than females did, as did business management majors in comparison to their education or humanities counterparts (table 14), it is possible that by excluding cases that were missing on EDPCTR, data presented in table 14 could have been overestimated. Furthermore, respondents missing on EDPCTR were less likely than their counterparts to have borrowed $25,000 or more (19 vs. 26 percent); thus, they logically could potentially bring down the numbers reported in table 14, had their true values been captured by EDPCTR. However, this possibility of overestimating might have been more or less offset by the fact that respondents with missing values for EDPCTR were more likely than those with known values to have graduated from private, not-for-profit, non-doctoral institutions (24 vs. 19 percent) and less likely to have finished up their bachelor’s degree at public non-doctoral institutions (17 vs. 20 percent), which means a possible downward bias since the former had higher median debt burden than the latter did (8 vs. 6, table 14).

Because EDPCTR is a critical variable, to further illustrate the potential impact of the missing cases on the statistics reported in table 14, the upper and lower bounds of the possible bias were examined. First, if one assumes that all missing cases happened to be males who had majored in business and management, had borrowed less than $25,000 in total for their undergraduate education, and had graduated from public non-doctoral institutions, then by assigning them each the average value on EDPCTR among respondents with those same characteristics and recalculating the median for the entire sample, one would get the lower bound of the estimate for the true median debt burden in 2001 among all bachelor’s degree recipients of year 1999-2000 who were in repayment at the time. Likewise, the upper bound could be obtained by defining all missing cases as females who had majored in education or humanities majors, had a total undergraduate debt of $25,000 or above, and had completed their bachelor’s degree at a private, not-for-profit non-doctoral institution. Using these projections, the lowest and highest bounds for the median estimate of 6.9 reported in table 14 are 5.0 and 9.8. That is, the maximum possible bias could produce estimates between 5 and 9.8 for the median debt burden.

The final variable requiring a bias analysis is NBMARR (marital status at NPSAS interview) from B&B:2000/01, which had an item response rate of 77 percent. Like ALLOWER, NBMARR was used only once in the report; it was presented as a column variable in table 18. Therefore, a bias analysis identical to that for ALLOWER was done by comparing nonrespondents with respondents for all the row variables used in the table (table 18): TOTDEBT (total amount borrowed for undergraduate education), CEANNERN (annual salary income), and EDPCTR (debt burden). Nonrespondents differed from respondents only in two respects. First, respondents with missing values for NBMARR were slightly more likely to have borrowed a total of $15,000–19,999 (13 vs. 11 percent). However, borrowers at this level were as likely as the average students in the cohort to be married at the time of the NPSAS interview (22 vs. 27 percent). Second, nonrespondents were more likely than respondents to have a debt burden averaging between 13 and 16 percent of their monthly income. Nonetheless, as shown in table 18, debt burden was not associated with the likelihood of being married at the time of bachelor’s degree receipt. Thus, it is unlikely that those with missing values on NBMARR would have caused much, if any, bias on the relevant statistics reported in this study.

next section