Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 U.S. History Assessment, by Item and Block: 2006
NAEP Technical DocumentationRange of response codes, percent exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 U.S. history assessment, by item and block: 2006
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
H3 H035801 1–3 500 92.00 † 0.80
H035901 1–3 600 92.00 † 0.90
H035902 1–3 500 86.00 † 0.82
H036101 1–3 500 90.00 † 0.91
H036402 1–4 500 78.00 † 0.82
H4 H059001 1–3 500 88.00 † 0.90
H059201 1–3 500 76.00 † 0.79
H059701 1–3 500 92.00 † 0.92
H059801 1–3 500 87.00 † 0.90
H060201 1–3 500 83.00 † 0.85
H5 H038103 1–4 400 83.00 † 0.71
H038301 1–3 500 93.00 † 0.93
H038601 1–3 500 88.00 † 0.76
H038702 1–3 500 88.00 † 0.82
H039001 1–3 500 95.00 † 0.93
H6 H069001 1–3 500 74.00 † 0.76
H069401 1–3 500 92.00 † 0.92
H069601 1–3 500 89.00 † 0.84
H069801 1–4 500 87.00 † 0.90
H070001 1–2 500 96.00 0.91 †
H7 H057501 1–3 600 97.00 † 0.95
H057701 1–3 500 87.00 † 0.81
H057801 1–3 500 83.00 † 0.83
H058601 1–3 500 85.00 † 0.82
H058701 1–3 500 93.00 † 0.92
H8 H034101 1–4 600 88.00 † 0.81
H034401 1–3 500 96.00 † 0.95
H034501 1–2 600 99.00 0.97 †
H034702 1–3 500 94.00 † 0.91
H035001 1–3 500 82.00 † 0.84
H035101 1–3 500 92.00 † 0.91
H9 H060701 1–3 500 92.00 † 0.91
H061501 1–3 600 92.00 † 0.91
H061601 1–3 500 91.00 † 0.86
H061801 1–3 500 96.00 † 0.93
H10 H042201 1–3 500 95.00 † 0.85
H042801 1–4 500 89.00 † 0.86
H042902 1–3 500 92.00 † 0.93
H043001 1–3 500 89.00 † 0.82
H043101 1–3 500 94.00 † 0.93
H11 H070401 1–3 500 82.00 † 0.81
H070701 1–4 500 95.00 † 0.96
H070801 1–3 500 88.00 † 0.83
H071001 1–4 600 84.00 † 0.81
H071601 1–3 500 81.00 † 0.79
H12 H071901 1–3 500 89.00 † 0.91
H072601 1–3 500 92.00 † 0.82
H072701 1–4 500 95.00 † 0.97
H073001 1–3 500 87.00 † 0.90
H073101 1–3 500 81.00 † 0.83
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 U.S. History Assessment.
Last updated 19 November 2009 (GF)

Printer-friendly Version