Understanding the 2009 Reading Trend Study NAEP frameworks provide the basis for the content of the assessments in each subject area, and describe the types of questions that should be included, as well as how those questions should be designed and scored. The NAEP frameworks are developed under the guidance of the National Assessment Governing Board. Frameworks are periodically updated or redeveloped in order to reflect current educational practice, such as changes in standards or coursework.
A new framework was developed for the 2009 NAEP reading assessment at grades 4, 8, and 12. The framework (2055K PDF) describes how reading is defined for the 2009 assessment and how this differs from the previous reading framework. The previous reading framework was first implemented in 1992 and was used for subsequent assessments through 2007 (or 2005 at grade 12). Because these assessments are based on the same framework, and therefore on a common definition of and approach to assessing reading comprehension, the results can be directly compared and reported as a trend line.
Past NAEP practice has been to start a new trend line when a new framework is introduced. However, special analyses were conducted in 2009 to determine if the results from the 2009 reading assessment could be compared to results from earlier years despite being based on a new framework.
The first step was to conduct a content alignment study to closely examine and compare the new (2009) and old (1992–2007) reading frameworks. Alignment studies are often used to help determine the extent to which two assessments are similar with respect to their purpose, characteristics, and content. A panel of content experts, such as reading teachers and teacher educators, looked at reading passages and questions from both the old and new assessments and judged how well they aligned with the specifications of each framework. It was determined that the old and new reading passages and questions were sufficiently similar to continue to the next stage of the special analyses: a reading trend study in 2009.
The purpose of the 2009 reading trend study was to compare results based on the 2009 and 2007 reading assessment instruments. Trend studies, also referred to as bridge studies, have previously been used in NAEP to evaluate the impact of changes to the assessment on the comparability of scores. For instance, results of a 2004 bridge study demonstrated that trend lines could be continued after a number of changes were made to the long-term trend assessments in reading and mathematics.
In the 2009 reading trend study, students were randomly assigned to take the old (2007) assessment, the new (2009) assessment, or a specially designed mixed assessment that contained material from both the old and new assessments. By administering both the old and new assessments in 2009, and by having some students answer questions from both assessments, it was possible to examine empirically the relationship between the old and new assessments. If analyses showed that the old and new assessments were similar, then it would be possible to compare the 2009 results directly to those from previous years. The special analyses into the relationship between the old and new assessments focused on three main questions:
Overall, the results of the special analyses suggested that the old and new assessments were similar in terms of their item and scale characteristics and the results they produced for important demographic groups of students. It was determined that the results of the 2009 reading assessment could still be compared to those from earlier assessment years, thereby maintaining the trend lines established in 1992. The results reported for 2009 are based on the total pool of questions administered to students in 2009—that is, the reading scales are based on the performance of students who took the old, new, and mixed assessments in 2009.
Although the reading trend lines are maintained, the implementation of the new framework for 2009 and for future assessment years meant that some of the differences between the old and new frameworks had to be addressed in the analysis of the 2009 data. Two such changes are discussed here:
|1992–2007 Framework||2009 Framework|
|Context/Subscale||Grade 4||Grade 8||Grade 12||Grade 4||Grade 8||Grade 12|
|† Not applicable. Reading to perform a task is not assessed.|
|SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1992–2009 Reading Assessments.|
The reading composite scale results reported for 2009 are based on the 2009 framework specifications. Analyses affirmed that the composite scales reported for grades 4, 8, and 12 in 2009 could be validly compared to composite results from previous years despite the changes in framework specifications.
For more information on NAEP scaling procedures in general, read technical documentation on the
estimation of scale scores.