Significance Tests: Definition

Tests for statistical significance indicate whether observed differences between assessment results occur because of sampling error or chance. Such "insignificant" results should be ignored because they do not reflect real differences. ("Significance" here does not imply any judgment about absolute magnitude or educational relevance. It refers only to the statistical nature of the difference and indicates the difference is worth taking note of.)

With your report of interest selected, click the Significance Test tab. You first need to decide the variable you want to test and the criterion by which you want to test the variable. You will compare or “look across” the criterion's range of values, so it must have more than one value. For example, you can look across years or across jurisdictions for a variable. You can look across the values within a variable – such as "male" and "female" within "gender." Once the primary criterion is chosen, all other criteria must be restricted to a single value.

Significance Test window.

1. In the Significance Test window select from Between Jurisdictions, Within Variables, or Across years.

To get a map of the U.S. showing states compared to the nation or to other states, select output type Map instead of Table. you will not be able to select score details because scores or percentages are seen when you mouse over the jurisdictions you tested for significance.

If you want to change the focal jurisdiction, click on your choice.

Select the statistic from the drop-down menu, and when you have the map you want, click Done. As with the Charts, you may edit again with Click here to edit this map or click anywhere in the map area to be returned to edit mode. Be sure to click Done when you have are finished so that the map saves to the Export Reports menu.

2. Provide a name for your significance test (25 or fewer alpha numeric characters and dash and underscore).

3. Choose whether your data will be formatted as a Table or a Map.

4. Select show score details to include the numerical value for each jurisdiction. You will see the matrix with the tests including the difference, P-value, and family size.

Note: Much information is included on the significance test matrix. All comparisons are independent tests with an alpha level of 0.05 adjusted for multiple pairwise comparisons according to the False Discovery Rate (FDR) procedure. For comparisons between two jurisdictions, a dependent test is performed for cases where one jurisdiction is contained in the other.

Note: The family size is the number of significance tests that are performed simultaneously. The larger the family size, the more the significance level for each individual test is reduced in order to reduce the chance of claims about significant differences that are due to chance alone.

Note: For more information on this topic, including comparing two groups, multiple comparisons, t-tests, the validity of inferences drawn from NAEP data, the Bonferroni procedure, the FDR procedure in NAEP, and related topics, see Drawing Inferences From NAEP Results at on this website.