Treatment of Items in NAEP Assessments

Items with responses that do not appear to fit the item response models used to describe the data may be treated in special ways. Rarely, items with responses that do not fit the models may be deleted from NAEP scales. For instance, dichotomous items with responses that are non-monotonic would be deleted from NAEP scales. These are items where the likelihood of a correct response is lower for students who do well on the other items than for students who do poorly on the other items. Other items that might be deleted from NAEP scales are those that can not be modified to fit the item response models and that have empirical and theoretical item response functions that vary a great deal.

Information about the way specific items are treated in NAEP scales is carefully tracked. Related items might be combined into "cluster items" and treated as a single item during scaling. Related items can be identified by test developers when items are written or scored, or by analysis staff when item responses are prepared for scaling or when items are scaled. Combining two or more items into a single item ameliorates dependencies among the items, making the assumption of local independence more realistic.

Polytomous items with categories of response data that do not fit the generalized partial credit model when empirical and theoretical item response functions are compared may fit the model more closely if categories of response data are combined. Therefore, some categories of polytomous items might be recorded to reflect the combination of response categories.

Finally, when items that were administered in more than one assessment year have responses that differ for those years, items might be scaled separately across assessment years. These items are most often identified by empirical item response functions for the different assessment years that differ markedly from one another and from the overall theoretical item response function.

Printer-friendly Version