Mapping State Proficiency Standards Onto NAEP Scales: Variation and Change in State Standards for Reading and Mathematics, 2005-2009

August 10, 2011

Author: Victor Bandeira de Mello

Download the complete report in a PDF file for viewing and printing. (1959K PDF)

---

Executive Summary

State-level National Assessment of Educational Progress (NAEP) results are an important resource for policymakers and other stakeholders responsible for making sense of and acting on state assessment results. Since 2003, the National Center for Education Statistics (NCES) has supported research that focuses on comparing NAEP and state proficiency standards. By showing where states’ standards lie on the NAEP scale, the mapping analyses offer several important contributions. First, they allow each state to compare the stringency of its criteria for proficiency with that of other states. Second, mapping analyses inform a state whether the rigor of its standards, as represented by the NAEP scale equivalent of the state’s standard, changed over time. (A state’s NAEP scale equivalent is the score on the NAEP scale at which the percentage of students in a state’s NAEP sample who score at or above that value matches the percentage of students in the state who score proficient or higher on the state assessment.) Significant differences in NAEP scale equivalents might reflect changes in state assessments and standards or changes in policies or practices that occurred between the years. Finally, when key aspects of a state’s assessment or standards remain the same, these mapping analyses allow NAEP to substantiate state-reported changes in student achievement.

The following are the research questions and the key findings regarding state proficiency standards, as they are measured on the NAEP scale.

How do states’ 2009 standards for proficient performance compare with one another when mapped onto the NAEP scale?

There is wide variation among state proficiency standards.

In 2009, as in 2003, 2005, and 2007, using NAEP as common metric, standards for proficient performance in reading and mathematics varied across states in terms of the levels of achievement required. For example, for grade 4 reading, the difference in the level required for proficient performance between the five states with the highest standards and the five with the lowest standards was comparable to the difference between Basic and Proficient performance on NAEP. The results for reading at grade 8 and mathematics in both grades were similar.

Most states’ proficiency standards are at or below NAEP’s definition of Basic performance.

In grade 4 reading, 35 of the 50 states included in the analysis set standards for proficiency (as measured on the NAEP scale) that were lower than the scale score for Basic performance on NAEP and another 15 were in the NAEP Basic range. In grade 8 reading, 16 of 50 states set standards that were lower than the cut-point for Basic performance on NAEP and another 34 were in the NAEP Basic range.

In grade 4 mathematics, seven of the 50 states included in the analysis set standards for proficiency (as measured on the NAEP scale) that were lower than the Basic performance on NAEP, 42 were in the NAEP Basic range, and one in the Proficient range. In grade 8 mathematics, 12 of 49 states included in the analysis set standards that were lower than the Basic performance on NAEP, 36 were in the NAEP Basic range, and one in the Proficient range.

How do the 2009 NAEP scale equivalents of state standards compare with those estimated for 2007 and 2005?

While NAEP adopted a revised reading framework in 2009, comparability with earlier assessments was maintained. During the same period, however, some states made changes in their assessments—changes substantial enough that the states indicated comparisons between scores of successive administrations were not possible.

Comparisons between the 2009 mapping results and the 2005 and 2007 mapping results in reading and mathematics at grades 4 and 8 were conducted separately for states that made changes in their testing systems and for those that made no such changes.

For those states that made substantive changes in their assessments between 2007 and 2009 most moved toward more rigorous standards as measured by NAEP.

When examined across grades 4 and 8 for both reading and mathematics, of the 34 cases where states reported changes in their assessments (9 states in reading and 8 states in mathematics), the rigor of the standards increased in 21 cases, 8 showed no change in their standards, and in 5 cases the rigor of their standards (as measured by NAEP scale equivalents) decreased.

For those states that made substantive changes in their assessments between 2005 and 2009, changes in the rigor of states’ standards as measured by NAEP were mixed but showed more decreases than increases in the rigor of their standards.

When examined across grades 4 and 8 for both reading and mathematics, of the 79 cases where states reported changes in their assessments (17 states in grade 4 reading, 20 in grade 8 reading, 19 in grade 4 mathematics, and 23 in grade 8 mathematics), the rigor of the standards increased in 25 cases, 14 showed no change in their standards, and in 40 cases the rigor of their standards (as measured by NAEP scale equivalents) decreased.

Does NAEP corroborate a state’s changes in the proportion of students meeting the state’s standard for proficiency from 2007 to 2009? From 2005 to 2009?

Changes in the proportion of students meeting states’ standards for proficiency between 2007 and 2009 are not corroborated by the proportion of students meeting proficiency, as measured by NAEP, in at least half of the states in the comparison sample.

In both subjects, changes in achievement between 2007 and 2009 on the state assessments do not agree with changes as measured by NAEP in the same period in at least half of the 40 states with comparable assessments in both years (22 to 26 depending on the subject and grade). In other words, the state assessment and NAEP reports show changes in percentages of students meeting the state’s standard that are significantly different from each other. In most cases (17 to 22 depending on the subject and grade), states’ results show more positive changes than NAEP results (larger gains or smaller losses).

Results of comparisons between changes in the proportion of students meeting states’ standards for proficiency between 2005 and 2009 and the proportion of students meeting proficiency, as measured by NAEP, were mixed.

The changes from 2005 to 2009 were mixed. For the two subject areas and grade levels, 16 to 18 states have comparable assessments in 2005 and 2009. In reading at grade 4 and in mathematics at grade 8, the changes in the proportion of students meeting the state’s proficiency standard are not significantly different from the changes in the proportion meeting the standard as measured by NAEP in more than half of the states (10 of 17 states and 10 of 16 states, respectively). However, these changes are different from each other in more than half of the states in reading at grade 8 (14 of 18 states) and mathematics at grade 4 (10 of 16 states). In most cases, states’ results showed more positive changes (12 of 14 and 8 of 10 states, respectively).

---

Download the complete report in a PDF file for viewing and printing. (1959K PDF)

NCES 2011-458 See the entry in the NCES database for contact and ordering information, and for links to similar topics.

Suggested Citation
Bandeira de Mello, V. (2011), Mapping State Proficiency Standards Onto NAEP Scales: Variation and Change in State Standards for Reading and Mathematics 2005-2009 (NCES 2011-458). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC: Government Printing Office.