NAEP Technical DocumentationThe Bonferroni Procedure

One general method for controlling error rates in multiple comparisons is based on the Bonferroni inequality. In this method, the Bonferroni inequality is applied and α is divided by the family size, n. For example, with a family size of four, α = 0.05/4 = 0.0125, and using α, the combined probability of one or more errors in the four comparisons remains controlled at less than or equal to 0.05. Note that dividing the probability by n is not the same as multiplying the critical value or the confidence band by n. Indeed, in moving from a family size of 1 to 4, we increase the critical value only from 1.960 to 2.498, a 27.4 percent increase. Doubling the family size again, to 8, increases the critical value to 2.735, an additional 9.5 percent increase. To double the initial critical value to 3.92, the family size would have to be increased to 564.

The power of the tests thus depends on the number of comparisons planned. There may be cases for which, before the data are seen, it is determined that only certain comparisons will be conducted. As an example, with the five groups above, interest might lie only in comparing the first group with each of the others (family size of four), rather than comparing all possible pairs of groups (family size of ten). This means that some possibly significant differences will not be found or discussed, but the planned comparisons will have greater power to identify real differences when they occur.

Last updated 11 November 2007 (TS)

Printer-friendly Version