Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Mathematics Combined National and State Main Assessment: 2003
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 mathematics combined national and state main assessment: 2003
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
M3 M019901 1–3 1,600 99 † 0.98
M020101 1–2 1,600 99 0.97 †
M047301 1–4 1,600 99 † 0.92
M075401 1–3 1,600 92 † 0.93
M075601 1–3 1,600 96 † 0.97
M076001 1–5 1,500 90 † 0.93
M4 M046001 1–5 1,600 99 † 0.98
M046601 1–4 1,600 98 † 0.97
M046801 1–5 1,600 100 † 0.99
M046901 1–5 1,700 99 † 0.98
M053001 1–2 1,200 96 0.93 †
M066601 1–3 1,600 94 † 0.95
M067201 1–3 1,600 84 † 0.87
M068003 1–3 1,600 98 † 0.98
M068005 1–3 1,600 97 † 0.97
M068006 1–3 1,500 91 † 0.93
M068008 1–3 1,600 92 † 0.93
M086301 1–5 1,100 97 † 0.99
M093601 1–3 1,400 98 † 0.98
M5 M019701 1–2 1,700 99 0.98 †
M020001 1–2 1,600 99 0.98 †
M047901 1–3 1,500 99 † 0.98
M053101 1–5 1,500 89 † 0.94
M066301 1–3 1,600 98 † 0.95
M066501 1–3 1,600 92 † 0.94
M067901 1–3 1,600 97 † 0.97
M085701 1–3 1,600 98 † 0.98
M085901 1–3 1,600 97 † 0.97
M6 M019801 1–3 1,600 97 † 0.94
M020201 1–2 1,600 97 0.79 †
M020301 1–4 1,600 99 † 0.99
M020401 1–2 1,700 99 0.98 †
M020501 1–2 1,600 98 0.97 †
M020801 1–6 1,600 96 † 0.99
M021201 1–3 1,600 98 † 0.98
M052201 1–5 1,500 91 † 0.96
M7 M013031 1–4 1,600 99 † 0.99
M021101 1–3 1,600 93 † 0.95
M051301 1–2 1,600 99 0.97 †
M073401 1–3 1,600 95 † 0.95
M092001 1–5 1,500 84 † 0.95
M8 M020901 1–2 1,700 92 0.85 †
M021001 1–2 1,700 98 0.97 †
M073601 1–5 1,400 90 † 0.97
M092401 1–3 1,700 96 † 0.93
M092601 1–3 1,600 96 † 0.96
M9 M013131 1–2 1,700 98 0.96 †
M052401 1–2 1,600 92 0.84 †
M072901 1–3 1,600 92 † 0.93
M073501 1–3 1,600 95 † 0.94
M075301 1–3 1,600 94 † 0.94
M075801 1–3 1,500 85 † 0.89
M093801 1–5 1,300 93 † 0.98
M10 M021301 1–2 1,700 95 0.91 †
M021302 1–2 1,700 98 0.97 †
M051201 1–2 1,700 99 0.98 †
M051601 1–2 1,700 99 0.97 †
M052101 1–3 1,600 97 † 0.96
M052901 1–2 1,700 93 0.86 †
M067501 1–5 1,400 92 † 0.92
M068201 1–5 1,500 92 † 0.96
M093501 1–3 1,600 95 † 0.96
M11 M106301 1–2 1,600 99 0.98 †
M106901 1–2 1,600 92 0.84 †
M107501 1–2 1,600 99 0.98 †
M12 M110201 1–2 1,700 98 0.96 †
M110501 1–3 1,600 94 † 0.95
M111801 1–3 1,500 89 † 0.93
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Mathematics Assessment.
Last updated 25 March 2009 (GF)

Printer-friendly Version