Constructed-Response Interrater Reliability

			Reliability Statistics for Items Rescored for Current-Year Rater Consistency Items from the Previous Assessment That Are Rescored During the Current Assessment

During the scoring of student responses, responses to constructed-response items are rescored by a second rater for one of two reasons:

• First, responses from the current assessment are rescored to determine how reliably the current
  raters are scoring responses to specific items. 
   
• Second, the responses to items given both in the current year and in previous years are rescored. This second type of rescoring indicates whether the current raters differ in their rating from the raters who had scored the same responses in previous years.

The statistics calculated for both of these purposes are the percentage of exact agreement, the intraclass correlation, Cohen's Kappa (Cohen, 1968), and the Pearson product-moment correlation between the scores for the first and second raters. These measures are summarized in Zwick (1988), Kaplan and Johnson (1992), and Abedi (1996). Each measure has advantages and disadvantages for use in different situations. Agreement percentages vary significantly across items. On a simple two-point mathematics item, agreement should approach 100 percent. On the other hand, when scoring a complex six-point writing constructed response item, an agreement of 60 percent would be considered an acceptable result. The trend year agreement percentage should approximate the interrater agreement from prior NAEP administration. The trend year agreement should be within 8 percent of the prior year interrater agreement for two- and three-point items and within 10 percent of the prior year interrater agreement for four- to six-point items.

Cohen’s Kappa quantifies another measure of reliability between groups of scorers and accounts for agreement due to chance. Kappa statistics should be higher than 0.7 for two- and three-point items and higher than 0.6 for four- to six-point items. Items with reliability statistics considered to be too low prompt investigation into the low rater agreement. The aforementioned criteria were created with NAEPs unique assessment design in mind. The percentage exact agreement for all constructed-response items, Cohen's Kappa for dichotomously scored constructed-response items, and the intraclass correlation for polytomously scored constructed-response items are provided for every NAEP assessment.

Printer-friendly Version