Skip Navigation
PEDAR: Research Methodology Debt Burden - A Comparison of 1992–93 and 1999–2000 Bachelor’s Degree Recipients a Year After Graduating
Baccalaureate and Beyond Longitudinal Study
Overall Response Rates
Weight Variables
Accuracy of Estimates
Item Response Rates and Bias Analysis
Data Analysis System
Statistical Procedures
Differences Between Means
Executive Summary
Full Report (PDF)
Executive Summary (PDF)
 Statistical Procedures - Differences Between Means

The descriptive comparisons in this report were tested using Student’s t statistic. Differences between estimates are tested against the probability of a Type I error,3 or significance level. The significance levels were determined by calculating the Student’s t values for the differences between each pair of means or proportions and comparing these with published tables of significance levels for two-tailed hypothesis testing (p<0.05).

Student’s t values may be computed to test the difference between estimates with the following formula:

equation 1

where E1 and E2 are the estimates to be compared and se1 and se2 are their corresponding standard errors. This formula is valid only for independent estimates. When estimates are not independent, a covariance term must be added to the formula:

equation 2

where r is the correlation between the two estimates.4 This formula is used when comparing two percentages from a distribution that adds to 100. If the comparison is between the mean of a subgroup and the mean of the total group, the following formula is used:

equation 3

where p is the proportion of the total group contained in the subgroup.5 The estimates, standard errors, and correlations can all be obtained from the DAS.

There are hazards in using statistical tests for each comparison. First, comparisons based on large t statistics may appear to merit special attention. This can be misleading since the magnitude of the t statistic is related not only to the observed differences in means or percentages but also to the number of respondents in the specific categories used for comparison. Hence, a small difference compared across a large number of respondents would produce a large t statistic.

A second hazard in using statistical tests is the possibility that one can report a “false positive” or Type I error. In the case of a t statistic, this false positive would result when a difference measured with a particular sample showed a statistically significant difference when there is no difference in the underlying population. Statistical tests are designed to control this type of error, denoted by alpha. The alpha level of .05 selected for findings in this report indicates that a difference of a certain magnitude or larger would be produced no more than one time out of 20 when there was no actual difference in the quantities in the underlying population. When researchers test hypotheses that show t values below the .05 significance level, they treat this finding as rejecting the null hypothesis that there is no difference between the two quantities. Failing to reject the null hypothesis (i.e., finding no difference), however, does not necessarily imply that the values are the same or equivalent.