Skip to main content

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 U.S. History Assessment: 2001
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 U.S. history assessment: 2001
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
H3 H044301 1–4 600 78 † 0.89
H044501 1–4 600 90 † 0.96
H044702 1–3 600 94 † 0.94
H045102 1–3 600 92 † 0.96
H045301 1–3 600 94 † 0.96
H4 H045501 1–4 600 81 † 0.87
H045901 1–4 600 80 † 0.89
H046001 1–2 600 99 0.99 †
H046101 1–3 600 96 † 0.96
H046301 1–3 600 95 † 0.96
H5 H062001 1–3 600 84 † 0.86
H062201 1–3 600 92 † 0.94
H063101 1–3 600 87 † 0.91
H063401 1–4 600 87 † 0.95
H063601 1–3 600 89 † 0.94
H6 H048901 1–3 600 99 † 1.00
H049401 1–4 600 86 † 0.77
H049503 1–3 600 86 † 0.88
H049601 1–4 600 87 † 0.93
H049701 1–3 600 98 † 0.99
H7 H050101 1–4 600 82 † 0.91
H050201 1–4 600 82 † 0.91
H051002 1–3 600 99 † 0.99
H051101 1–3 600 87 † 0.89
H051102 1–3 600 82 † 0.90
H8 H051301 1–3 600 86 † 0.89
H052301 1–4 600 73 † 0.84
H052501 1–3 600 92 † 0.96
H052601 1–3 600 86 † 0.90
H052701 1–3 600 88 † 0.91
H9 H060701 1–3 600 94 † 0.97
H061501 1–3 600 82 † 0.86
H061601 1–3 600 91 † 0.91
H061801 1–3 600 96 † 0.96
H10 H042201 1–3 600 97 † 0.94
H042801 1–4 600 92 † 0.95
H042902 1–3 600 95 † 0.97
H043001 1–3 600 86 † 0.90
H043101 1–3 600 97 † 0.98
H11 H063801 1–3 600 82 † 0.80
H064001 1–3 600 88 † 0.88
H064101 1–3 600 89 † 0.93
H064401 1–4 600 82 † 0.91
H064901 1–3 600 84 † 0.86
H065101 1–3 600 79 † 0.80
H065201 1–3 600 78 † 0.85
H065301 1–3 600 86 † 0.92
H065401 1–4 600 88 † 0.95
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version