Skip to main content

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 8 U.S. History Assessment: 2001
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 8 U.S. history assessment: 2001
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
H3 H035801 1–3 600 92 † 0.77
H035901 1–3 600 94 † 0.95
H035902 1–3 600 84 † 0.77
H036101 1–3 600 86 † 0.87
H036402 1–4 600 66 † 0.73
H5 H038103 1–4 600 88 † 0.74
H038301 1–3 600 91 † 0.92
H038601 1–3 600 87 † 0.65
H038702 1–3 600 89 † 0.82
H039001 1–3 600 95 † 0.95
H6 H039401 1–3 600 89 † 0.83
H039901 1–4 600 89 † 0.90
H040001 1–3 600 93 † 0.91
H040103 1–3 600 81 † 0.78
H040201 1–3 600 91 † 0.91
H8 H034101 1–4 600 87 † 0.81
H034401 1–3 600 93 † 0.88
H034501 1–2 600 100 1.00 †
H034702 1–3 600 91 † 0.83
H035001 1–3 600 80 † 0.84
H035101 1–3 600 88 † 0.78
H10 H042201 1–3 600 99 † 0.94
H042801 1–4 600 89 † 0.87
H042902 1–3 600 94 † 0.95
H043001 1–3 600 85 † 0.77
H043101 1–3 600 94 † 0.93
H11 H043201 1–3 600 87 † 0.85
H043401 1–3 600 89 † 0.92
H043501 1–4 600 81 † 0.88
H043601 1–3 600 86 † 0.85
H043701 1–3 600 90 † 0.92
H043705 1–3 600 78 † 0.76
H044001 1–4 600 79 † 0.78
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version