The Setting of Achievement Levels

• Methods for NAEP standards setting
• More information about achievement levels

Methods for NAEP standards setting

Different methods were used to set achievement levels in the various NAEP subjects assessed:

1. The achievement levels in mathematics were set in 1992, according to the modified Angoff process described in Appendix G to The 1992 NAEP Technical Report. The achievement levels were validated in 1996, according to the modified Angoff process described in Appendix F to The 1996 NAEP Technical Report. The achievement levels in mathematics were corrected for an information weighting error in 1996 as described in Appendix H to this technical report. Note that the 1992 and 1994 technical reports are not online; please Contact NAEP, specifying the publication and appendix needed.
2. The achievement levels in reading were set in 1992 and validated in 1994, according to the modified Angoff processes described in Appendix H to& The 1992 NAEP Technical Report, and in Appendix F to The 1994 NAEP Technical Report. The achievement levels in reading were also corrected for an information weighting error in 1996 as described in Appendix H to The 1996 NAEP Technical Report.
3. The achievement levels for the 1994 U.S. history and geography assessments were set according to the modified Angoff process described in Appendix G to The 1992 NAEP Technical Report.
4. Achievement level descriptions for the 1996 science assessment were developed for cut scores set by the National Assessment Governing Board, as described in Appendix G of The 1996 NAEP Technical Report.
5. The achievement levels for the 1998 civics and writing assessments were set according to the modified Angoff process (with Reckase chart feedback that encouraged the ratings to be consistent with the IRT scales) as described in Appendix J to The NAEP 1998 Technical Report.
6. The achievement levels for the 2005 grade 12 mathematics assessment were developed using a modified bookmark method with subscale feedback, as described in Developing Achievement Levels on the 2005 National Assessment of Educational Progress in Grade Twelve Mathematics: Process Report in the Governing Board publications about achievement levels.
7. The achievement levels for the 2007 grade 12 economics assessment were developed using a bookmark method, as described in Developing Achievement Levels on the 2006 National Assessment of Educational Progress in Grade Twelve Economics: Process Report in Governing Board publications.
8. The achievement levels for the 2009 science assessment were developed using a bookmark method, as described in Developing Achievement Levels on the 2009 National Assessment of Educational Progress in Science for Grades Four, Eight, and Twelve: Process Report and also the Technical Report, in Governing Board publications.
9. The achievement levels for the 2011 writing assessment were developed using a body of work method, as forecast in Developing Achievement Levels on the National Assessment of Educational Progress for Writing Grades 8 and 12 in 2011 and Grade 4 in 2013 in Governing Board publications.

All of the variations in the process of standard setting begin with developing a description of what students should know and be able to do to qualify for performance at each of the three NAEP achievement levels. The panelists then judge whether students who just meet the requirements would have the knowledge required to answer items on NAEP correctly. The cut score is set to represent the minimal performance required for each level of achievement level. The Basic cut score represents the minimal performance to meet the requirements described for that level. The Basic achievement level extends to the cut score for Proficient.

All of the variations in the process of standard setting include judges who are expert in the subject matter, including not only classroom teachers in the subject and grade being assessed, but also other educators (such as college faculty and curriculum directors) and representatives of the general public who are trained in the content area and have knowledge of the skills and educational requirements for students at the grade levels assessed by NAEP.

The result of each process of standard setting is the identification on the NAEP scale for each subject the score that corresponds to the lower boundary of each achievement level--the cut score. After the cut scores are set, the judges also select exemplar items that are good examples of the kinds of knowledge and skills that students in each achievement level can answer correctly.

More information about achievement levels

The NAEP technical documentation describes the achievement levels for each subject in the first chapter, Analysis and Scaling, in the section about Describing NAEP Scale Score Distributions.

Current legislation requires that NAEP achievement levels be considered trial; read about their status. 

Top