Skip to main content

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1998 That Were Rescored in 2000, by Block and Item, Grade 4 Reading Assessment: 2000
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1998 that were rescored in 2000, by block and item, grade 4 reading assessment: 2000
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa> Intraclass correlation
R3 R017001 1–2 600 87 0.71 †
R017003 1–3 600 71 † 0.74
R017004 1–2 600 87 0.68 †
R017006 1–2 600 92 0.78 †
R017007 1–4 600 75 † 0.86
R017009 1–3 600 83 † 0.73
R4 R012102 1–2 600 96 0.93 †
R012104 1–2 600 94 0.87 †
R012106 1–2 600 85 0.70 †
R012108 1–2 600 94 0.85 †
R012109 1–2 600 94 0.82 †
R012111 1–4 600 86 † 0.88
R012112 1–2 600 92 0.81 †
R5 R012601 1–2 600 89 0.69 †
R012604 1–2 600 93 0.81 †
R012607 1–4 600 81 † 0.83
R012611 1–2 600 85 0.70 †
R7 R012702 1–2 600 92 0.79 †
R012703 1–2 600 89 0.76 †
R012705 1–2 600 91 0.68 †
R012706 1–2 600 80 0.49 †
R012708 1–4 600 80 † 0.77
R012710 1–2 600 92 0.80 †
R8 R015702 1–3 600 88 † 0.81
R015703 1–3 600 88 † 0.83
R015704 1–3 600 79 † 0.81
R015705 1–3 600 94 † 0.94
R015707 1–4 600 92 † 0.91
R015709 1–3 600 85 † 0.83
R9 R015802 1–2 600 91 0.79 †
R015803 1–3 600 87 † 0.81
R015804 1–4 600 74 † 0.73
R015806 1–3 600 84 † 0.80
R015807 1–3 600 76 † 0.73
R015809 1–3 600 87 † 0.83
R10 R012503 1–2 600 88 0.77 †
R012504 1–2 600 98 0.96 †
R012506 1–2 600 90 0.79 †
R012508 1–2 600 97 0.94 †
R012511 1–2 600 94 0.85 †
R012512 1–4 600 86 † 0.92
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Reading Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version