Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 12 Reading National Main Assessment: 2005
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2005, by block and item, grade 12 reading national main assessment: 2005
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
R3 R017101 1–2 2,000 85 0.69 †
R017102 1–3 600 86 † 0.84
R017104 1–3 600 89 † 0.86
R017105 1–4 400 72 † 0.78
R017107 1–3 400 86 † 0.88
R017108 1–2 400 96 0.87 †
R017110 1–3 400 91 † 0.91
R4 R013501 1–2 400 89 0.78 †
R013503 1–2 400 92 0.85 †
R013505 1–2 600 85 0.63 †
R013506 1–4 900 70 † 0.74
R013508 1–2 400 89 0.65 †
R013509 1–2 400 91 0.67 †
R5 R016301 1–3 800 73 † 0.71
R016302 1–3 400 77 † 0.67
R016303 1–3 900 78 † 0.71
R016305 1–3 1,300 70 † 0.72
R016306 1–3 600 79 † 0.78
R016307 1–3 600 79 † 0.75
R016308 1–4 600 80 † 0.80
R6 R013201 1–4 600 73 † 0.82
R013203 1–2 800 95 0.64 †
R013205 1–2 400 98 0.75 †
R013207 1–2 1,500 85 0.56 †
R013209 1–2 600 96 0.91 †
R013211 1–2 900 81 0.62 †
R013212 1–4 2,200 77 † 0.84
R7 R013701 1–2 800 88 0.76 †
R013702 1–2 400 84 0.68 †
R013704 1–2 800 87 0.56 †
R013706 1–4 1,300 73 † 0.72
R013708 1–2 600 88 0.74 †
R013710 1–2 800 90 0.69 †
R013712 1–2 900 80 0.58 †
R8 R053501 1–3 400 72 † 0.57
R053504 1–2 400 97 0.87 †
R053507 1–4 400 85 † 0.87
R053509 1–2 400 92 0.69 †
R10 R013403 1–4 600 94 † 0.94
R013405 1–2 600 92 0.84 †
R013406 1–4 800 80 † 0.90
R013407 1–2 400 88 0.75 †
R013409 1–2 600 89 0.75 †
R013411 1–2 600 92 0.82 †
R013413 1–2 400 83 0.59 †
R11 R019301 1–4 600 85 † 0.88
R019401 1–3 500 90 † 0.80
R019501 1–2 700 89 0.24 †
R019701 1–3 300 93 † 0.91
R020001 1–3 400 77 † 0.60
R020201 1–3 1,300 79 † 0.71
R13 R015503 1–2 1,300 84 0.62 †
R015505 1–2 800 88 0.75 †
R015507 1–4 900 75 † 0.84
R015509 1–2 900 86 0.65 †
R015512 1–2 800 86 0.72 †
R015514 1–4 900 79 † 0.83
R14 R016501 1–3 800 82 † 0.74
R016502 1–3 600 80 † 0.68
R016601 1–3 900 70 † 0.62
R016602 1–3 800 79 † 0.73
R016603 1–3 400 98 † 0.95
R016604 1–3 900 77 † 0.76
R016605 1–3 400 91 † 0.62
R016701 1–4 600 77 † 0.78
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.
Last updated 06 April 2009 (GF)

Printer-friendly Version