Skip Navigation
PEDAR: Research Methodology  The Road Lsss Traveled? Students Who Enroll in Multiple Institutions
Beginning Postsecondary Students Longitudinal Study
The 2001 Baccalaureate and Beyond Longitudinal Study
Accuracy of Estimates
Item Response Rates
Data Analaysis System
Statistical Procedures
Differences Between Means
Linear Trends
Multivariate Commonality Analysis
Missing Data and Adjusting for Complex Sampling Design
Interpreting the Results
Executive Summary
Full Report (PDF)
Executive Summary (PDF)
 Statistical Procedures: Differences Between Means

The descriptive comparisons were tested in this report using Student’s t statistic. Differences between estimates are tested against the probability of a Type I error,6 or significance level. The significance levels were determined by calculating the Student’s t values for the differences between each pair of means or proportions and comparing these with published tables of significance levels for two-tailed hypothesis testing.

Student’s t values may be computed to test the difference between estimates with the following formula:

equation 1(1)

where E1 and E2 are the estimates to be compared and se1 and se2 are their corresponding standard errors. This formula is valid only for independent estimates. When estimates are not independent, a covariance term must be added to the formula:

equation 2 (2)

where r is the correlation between the two estimates.7 This formula is used when comparing two percentages from a distribution that adds to 100. If the comparison is between the mean of a subgroup and the mean of the total group, the following formula is used:

equation 3(3)

where p is the proportion of the total group contained in the subgroup.8 The estimates, standard errors, and correlations can all be obtained from the DAS.

There are hazards in reporting statistical tests for each comparison. First, comparisons based on large t statistics may appear to merit special attention. This can be misleading since the magnitude of the t statistic is related not only to the observed differences in means or percentages but also to the number of respondents in the specific categories used for comparison. Hence, a small difference compared across a large number of respondents would produce a large t statistic.

A second hazard in reporting statistical tests is the possibility that one can report a “false positive” or Type I error. In the case of a t statistic, this false positive would result when a difference measured with a particular sample showed a statistically significant difference when there is no difference in the underlying population. Statistical tests are designed to control this type of error, denoted by alpha. The alpha level of .05 selected for findings in this report indicates that a difference of a certain magnitude or larger would be produced no more than one time out of twenty when there was no actual difference in the quantities in the underlying population. When we test hypotheses that show t values at the .05 level or smaller, we treat this finding as rejecting the null hypothesis that there is no difference between the two quantities. However, there are other cases when exercising additional caution is warranted. When there are significant results not indicated by any hypothesis being tested or when we test a large number of comparisons in a table, Type I errors cannot be ignored. For example, when making paired comparisons among different fields of study, the probability of a Type I error for these comparisons taken as a group is larger than the probability for a single comparison.

next section