Since the estimates in this report are based on a sample, observed differences between two estimates can reflect either of two possibilities: differences that exist in the population at large and are reflected in the sample, or differences due solely to the composition of the sample that do not reflect underlying population differences. To minimize the risk of erroneously interpreting differences due to sampling alone as signifying population differences (a Type I error), the statistical significance of differences between estimates was tested using a t-test. Statistical significance was determined by calculating t values for differences between pairs of means or proportions and comparing these with published values of t for two-tailed hypothesis testing, using a 5 percent probability of a Type I error (a significance level of .05).⁶

The t values may be computed to test the difference between estimates with the following formula:

(1)

where E₁ and E₂ are the estimates to be compared and se₁ and se₂ are their corresponding standard errors. Note that this formula is valid only for independent estimates. When estimates are not independent, a covariance term must be added to the formula:

(2)

where r is the correlation between the two variables.⁷ The denominator in this formula will be at its maximum when the two estimates are perfectly negatively correlated, that is, when r = –1. This means that a conservative dependent test may be conducted by using –1 for the correlation in this formula, or

(3)

The estimates and standard errors are obtained from the DAS.

There are hazards in reporting statistical tests for each comparison. First, comparisons based on large t statistics may appear to merit special attention. This can be misleading since the magnitude of the t statistic is related not only to the observed differences in means or percentages but also to the number of sample members in the specific categories used for comparison. Hence, a small difference compared across a large number of sample members would produce a large t statistic.

A second hazard in reporting statistical tests for each comparison occurs when making multiple comparisons between categories of an independent variable. For example, when making paired comparisons between different levels of income, the probability of a Type I error for these comparisons taken as a group is larger than the probability for a single comparison. When more than one difference between groups of related characteristics or “families” are tested for statistical significance, one must apply a standard that assures a level of significance for all of those comparisons taken together. One such procedure is known as the Bonferroni adjustment.

Comparisons were made in this report only when p < .05/k for a particular pairwise comparison, where that comparison was one of k tests within a family. This helps to assure both that the individual comparison would have p < .05 and that for k comparisons within a family of possible comparisons, the significance level for all the comparisons would sum to p < .05.⁸

For example, when comparing males and females, only one comparison is possible. In this family, k=1, and there is no need to adjust the significance level. When faculty members are divided into five racial/ethnic groups and all possible comparisons are made, then k=10 and the significance level for each test within this family of comparisons must be p < .05/10, or p < .005. The formula for calculating family size (k) is as follows:

(4)

where j is the number of categories for the variable being tested. For example, in the case of a variable with five categories such as race/ethnicity, one substitutes 5 for j in equation 4:

Different schools of thought exist on the application of the Bonferroni adjustment: while some would use an experiment-wise calculation of k, where all the dependent variables were considered simultaneously in selecting a critical value, here the calculation of k and the accompanying critical value were restricted to a single dependent variable at a time, since the Bonferroni adjustment is already a conservative strategy.