Understanding the 2009 Mathematics Trend Study at Grade 12

NAEP frameworks provide the basis for the content of the assessments in each subject area, and describe the types of questions that should be included, as well as how those questions should be designed and scored. The NAEP frameworks are developed under the guidance of the National Assessment Governing Board. Frameworks are periodically updated or redeveloped in order to reflect current educational practice, such as changes in standards or coursework.

A new framework was developed for the 2009 NAEP mathematics assessment at grade 12. The framework (1,654K PDF) describes how mathematics is defined for the 2009 assessment and how this differs from the previous mathematics framework. The previous mathematics framework at grade 12 was first implemented in 2005.

Past NAEP practice has been to start a new trend line when a new framework is introduced. For instance, a new trend line for grade 12 mathematics was started in 2005 when the previous framework was implemented; the 2005 results were not compared to results from earlier grade 12 mathematics assessments. However, special analyses were conducted in 2009 to determine if the results from the 2009 mathematics assessment could be compared to results from 2005 despite being based on a new framework.

The first step was to conduct a content alignment study to closely examine and compare the “new” (2009) and “old” (2005) mathematics frameworks. Alignment studies are often used to help determine the extent to which two assessments are similar with respect to their purpose, characteristics, and content. A panel of content experts, such as mathematics teachers and teacher educators, looked at questions from both the old and new assessments and judged how well they aligned with the specifications of each framework. It was determined that the old and new mathematics questions were sufficiently similar to continue to the next stage of the special analyses: a trend study in 2009.

The purpose of the 2009 mathematics trend study at grade 12 was to compare results based on the 2009 and 2005 mathematics assessment instruments at grade 12. Trend studies, also referred to as bridge studies, have previously been used in NAEP to evaluate the impact of changes to the assessment on the comparability of scores. For instance, results of a 2004 bridge study demonstrated that trend lines could be continued after a number of changes were made to the long-term trend assessments in reading and mathematics.

In the 2009 mathematics trend study at grade 12, students were randomly assigned to take the old (2005) assessment, the new (2009) assessment, or a specially designed “mixed” assessment that contained material from both the old and new assessments. By administering both the old and new assessments in 2009, and by having some students answer questions from both assessments, it was possible to examine empirically the relationship between the old and new assessments. If analyses showed that the old and new assessments were similar, then it would be possible to compare the 2009 results directly to those from previous years.

The special analyses into the relationship between the old and new assessments focused on three main questions:

1. How do the blocks of old and new assessment questions compare in terms of difficulty, student nonresponse rates, student ability to complete all questions in the block, and reliability?

2. What is the relationship between the old and new assessment scales?

3. Do the old and new assessments produce similar results (scale scores and percentages of students reaching the NAEP achievement levels) for major reporting groups?

Overall, the results of the special analyses suggested that the old and new assessments were similar in terms of their item and scale characteristics and the results they produced for important demographic groups of students. It was determined that the results of the 2009 mathematics assessment could still be compared to those from 2005. The results reported for 2009 are based on the total pool of questions administered to students in 2009—that is, the mathematics scale at grade 12 is based on the performance of students who took the old, new, and mixed assessments in 2009.

Results for 2009 are reported for the same four mathematics content-area subscales as in 2005: number properties and operations; measurement and geometry; data analysis, statistics, and probability; and algebra. In addition, results are reported for a composite NAEP mathematics scale. This composite scale is a weighted combination of the mathematics subscales, with the weights reflecting the percentage distribution of the grade 12 items across the mathematics content areas. The recommended item distribution is specified by the 2009 framework, and is identical to the percentage distribution in the 2005 framework.