Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Estimation of NAEP Score Scales → Checks for Violations of Assumptions → Comparisons of Empirical and Theoretical Item Response Functions → Items Showing Poor Item Fit → Treating Items as Different Across Assessment Years

NAEP Technical DocumentationTreating Items as Different Across Assessment Years

When the same items are presented in two assessment years, the empirical item response function (IRFs) for the two years can be compared. Normally, these curves differ somewhat due to the sampling of students for each of the two years. When the empirical curves differ dramatically, one cause might be a change in the meaning of the item due to instructional or societal changes across the years. This type of item is ordinarily treated as two different items—one for each of the assessment years.

This figure contains the plot of an item that was treated as two separate items for the 2000 and the 1996 grade 4 science assessments, when they were scaled together for the analysis of the 2000 assessment. The item is a multiple-choice item. The proportion of students who actually responded correctly, denoted by the two sets of triangles on the plot, differ in consistent ways from values predicted by the solid curve representing the theoretical estimates when data from both years were combined to estimate the Item Response Theory (IRT) parameters. The fact that one set of triangles representing the empirical proportions of students for one year is always above the theoretical (IRFs) and the other set of triangles is always below the theoretical IRF is an indication of this. As can be seen by connecting the sets of triangles into two curves, each appears to represent a logistic curve well.

In the plot, the horizontal axis represents the theta () scale and the vertical axis represents the probability of having a response fall in each category. The solid curve is the theoretical IRF based on the item parameter estimates and the equation for the three-parameter logistic model. Red inverted triangles represent 2000 grade 4 science assessment data; green triangles represent 1996 grade 4 science assessment data.

Dichotomous item (M018901) exhibiting different empirical item functions for different assessment years

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP).

Printer-friendly Version