The National Assessment of Educational Progress Authorization Act1 states the following:
Section 303 (e)
(1) Achievement Levels - The National Assessment Governing Board shall develop appropriate student achievement levels for each grade or age in each subject area to be tested under assessments authorized under this section, except the trend assessment described in subsection (b)(2)(F).
(2) Determination of Levels -
(i) be determined by--
The law requires that the achievement levels be used on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are "reasonable, valid, and informative to the public." Until that determination is made, the law requires the Commissioner and the National Assessment Governing Board to state clearly the trial status of the achievement levels in all NAEP reports.
The most recent congressionally mandated evaluation conducted by the National Academy of Sciences (NAS) relied on prior studies of achievement levels, rather than carrying out new evaluations, on the grounds that the process has not changed substantially since the initial concerns (these initial concerns being the subjectivity of the achievement level setting process) were identified. Instead, the NAS Panel studied the development of the 1996 science achievement levels. The NAS Panel basically concurred with earlier congressionally mandated studies. The Panel concluded that "NAEP's current achievement-level setting procedures remain fundamentally flawed. The judgment tasks are difficult and confusing; raters' judgments of different item types are internally inconsistent; appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results" (Pellegrino, Jones, and Mitchell 1998).
A proven alternative to the current process has not yet been identified. The Deputy Commissioner of Education Statistics and the Board continue to call on the research community to assist in finding ways to improve standard setting for reporting NAEP results. The NAS Panel accepted the continuing use of achievement levels in reporting NAEP results on a developmental basis, until such time as better procedures can be developed. Specifically, the NAS Panel concluded that "...tracking changes in the percentages of students performing at or above those cut scores (or, in fact, any selected cut scores) can be of use in describing changes in student performance over time" (Pellegrino, Jones, and Mitchell 1998). The National Assessment Governing Board urges all who are concerned about student performance levels to recognize that the use of these achievement levels is a developing process and is subject to various interpretations. The Board and the Deputy Commissioner believe that the achievement levels are useful for reporting trends in the educational achievement of students in the United States. However, based on the congressionally mandated evaluations so far, the Deputy Commissioner agrees with the National Academy's recommendation that caution needs to be exercised in the use of the current achievement levels. Therefore, the Deputy Commissioner concludes that these achievement levels should continue to be used on a trial basis and should continue to be interpreted and used with caution.