NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Main Assessment, by Item and Block: 2007

Range of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 mathematics combined national and state main assessment, by item and block: 2007
Block Item Range of response codes Sample size Percent agreement Cohen’s Kappa Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2007Mathematics Assessment.
M3 M149901 1–3 1,500 96.81 0.97
M150801 1–3 1,400 97.44 0.98
M150901 1–2 1,500 99.29 0.98
M151201 1–2 1,400 99.72 0.99
M4 M151501 1–3 1,500 94.89 0.96
M151601 1–3 1,400 94.54 0.94
M152602 1–5 1,500 96.84 0.98
M5 M019701 1–2 1,600 99.88 0.99
M020001 1–2 1,600 99.29 0.98
M047901 1–3 1,400 99.15 0.98
M053101 1–5 1,400 87.76 0.94
M066301 1–3 1,500 96.95 0.94
M066501 1–3 1,500 96.01 0.96
M067901 1–3 1,400 97.68 0.97
M085701 1–3 1,600 98.15 0.97
M085901 1–3 1,500 94.60 0.93
M6 M141301 1–4 1,500 96.08 0.97
M141601 1–2 1,400 85.58 0.69
M141901 1–5 1,400 87.70 0.93
M7 M144201 1–2 1,600 99.42 0.99
M144601 1–3 1,500 93.72 0.93
M144901 1–4 1,400 90.41 0.92
M145101 1–4 1,400 98.01 0.93
M8 M020901 1–2 1,400 90.94 0.82
M021001 1–2 1,600 99.05 0.98
M073601 1–5 1,400 84.91 0.94
M092401 1–3 1,600 97.52 0.96
M092601 1–3 1,500 97.69 0.97
M9 M013131 1–2 1,500 98.62 0.97
M052401 1–2 1,600 91.37 0.82
M072901 1–3 1,500 93.94 0.95
M073501 1–3 1,500 95.88 0.94
M075301 1–3 1,500 94.79 0.93
M075801 1–3 1,400 85.58 0.87
M093801 1–5 1,400 91.73 0.97
M10 M142801 1–2 1,400 99.65 0.99
M143201 1–2 1,500 98.94 0.98
M143501 1–5 1,400 84.09 0.92
M11 M106301 1–2 1,600 99.00 0.98
M106901 1–2 1,500 87.34 0.70
M107501 1–2 1,500 99.67 0.99
M12 M153201 1–3 1,600 88.95 0.88
M153601 1–2 1,500 98.59 0.97
M153901 1–3 1,400 86.52 0.89

Last updated 02 March 2010 (JL)