​​​NAEP Technical DocumentationConstructed-Response Interrater Reliability

			Reliability Statistics for Items Rescored for Current-Year Rater Consistency Items from the Previous Assessment That Are Rescored During the Current Assessment

During the scoring of student responses, some responses to constructed-response items are rescored by a second rater for one of two reasons:

• to determine how reliably the current raters are scoring responses to specific items; or 
  
• to determine whether the current raters differ in their rating from the raters who had scored the same responses in previous years.
  

The statistics calculated for both of these purposes are the percentage of exact agreement, the intraclass correlation, and Cohen's Kappa (Cohen 1968). These measures are summarized in Kaplan and Johnson (1992) and Abedi (1996). Each measure has advantages and disadvantages for use in different situations. Agreement percentages vary significantly across items. On a simple two-point mathematics item, agreement should approach 100 percent. On the other hand, when scoring a complex six-point writing constructed-response item, an agreement of 60 percent would be considered an acceptable result. The trend year agreement percentage should approximate the interrater agreement from the prior NAEP administration. The trend year agreement should be within 8 percent of the prior year interrater agreement for two- and three-point items and within 10 percent of the prior year interrater agreement for four- to six-point items. For more information about subject-specific targets for the interrater agreement, please refer to the TDW Scoring section.

Cohen’s Kappa quantifies the reliability between groups of scorers and accounts for agreement due to chance. Kappa statistics should be higher than 0.7 for two- and three-point items and higher than 0.6 for four- to six-point items. Items with reliability statistics considered to be too low prompt investigation into the low rater agreement. The aforementioned criteria were created with NAEP's unique assessment design in mind. The percentage of exact agreement for all constructed-response items, Cohen's Kappa for dichotomously scored constructed-response items, and the intraclass correlation for polytomously scored constructed-response items are provided for every NAEP assessment.

Printer-friendly Version