NAEP Technical DocumentationThe Status of Achievement Levels

(1) Achievement Levels - The National Assessment Governing Board shall develop appropriate student achievement levels for each grade or age in each subject area to be tested under assessments authorized under this section, except the trend assessment described in subsection (b)(2)(F).

(2) Determination of Levels -

1. In general - Such levels shall--

   (i) be determined by--

   (I) identifying the knowledge that can be measured and verified objectively using widely accepted professional assessment standards; and
   (II) developing achievement levels that are consistent with relevant widely accepted professional assessment standards and based on the appropriate level of subject matter knowledge for grade levels to be assessed, or the age of the students, as the case may be.
    
   
2. National consensus approach - After the determinations described in subparagraph (A), devising a national consensus approach.
    
   
3. Trial basis - The achievement levels shall be used on a trial basis until the Commissioner for Education Statistics determines, as a result of an evaluation under subsection (f), that such levels are reasonable, valid, and informative to the public.
    
   
4. Status - The Commissioner for Education Statistics and the Assessment Board shall ensure that reports using such levels on a trial basis do so in a manner that makes clear the status of such levels.
    
   
5. Updates - Such levels shall be updated as appropriate by the National Assessment Governing Board in consultation with the Commissioner for Education Statistics.
    
   

The law requires that the achievement levels be used on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are "reasonable, valid, and informative to the public." Until that determination is made, the law requires the Commissioner and the National Assessment Governing Board to state clearly the trial status of the achievement levels in all NAEP reports.

The most recent congressionally mandated evaluation conducted by the National Academy of Sciences (NAS) relied on prior studies of achievement levels, rather than carrying out new evaluations, on the grounds that the process has not changed substantially since the initial concerns (these initial concerns being the subjectivity of the achievement level setting process) were identified. Instead, the NAS Panel studied the development of the 1996 science achievement levels. The NAS Panel basically concurred with earlier congressionally mandated studies. The Panel concluded that "NAEP's current achievement-level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters' judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results" (Pellegrino, Jones, and Mitchell 1998).

A proven alternative to the current process has not yet been identified. The Deputy Commissioner of Education Statistics and the Board continue to call on the research community to assist in finding ways to improve standard setting for reporting NAEP results. The NAS Panel accepted the continuing use of achievement levels in reporting NAEP results on a developmental basis, until such time as better procedures can be developed. Specifically, the NAS Panel concluded that "...tracking changes in the percentages of students performing at or above those cut scores (or, in fact, any selected cut scores) can be of use in describing changes in student performance over time" (Pellegrino, Jones, and Mitchell 1998). The National Assessment Governing Board urges all who are concerned about student performance levels to recognize that the use of these achievement levels is a developing process and is subject to various interpretations. The Board and the Deputy Commissioner believe that the achievement levels are useful for reporting trends in the educational achievement of students in the United States. However, based on the congressionally mandated evaluations so far, the Deputy Commissioner agrees with the National Academy's recommendation that caution needs to be exercised in the use of the current achievement levels. Therefore, the Deputy Commissioner concludes that these achievement levels should continue to be used on a trial basis and should continue to be interpreted and used with caution.