Skip to main content

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2002, by Block and Item, Grade 12 Reading National Main Assessment: 2002
NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2002, by block and item, grade 12 reading national main assessment: 2002
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
R3 R017101 1–2 1,000 92 0.84 †
R017102 1–3 1,400 82 † 0.85
R017104 1–3 600 88 † 0.90
R017105 1–4 800 68 † 0.82
R017107 1–3 600 88 † 0.90
R017108 1–2 1,400 94 0.88 †
R017110 1–3 800 91 † 0.93
R4 R013501 1–2 2,200 90 0.80 †
R013503 1–2 900 94 0.87 †
R013505 1–2 1,400 87 0.69 †
R013506 1–4 2,600 76 † 0.75
R013508 1–2 900 90 0.78 †
R013509 1–2 1,900 90 0.79 †
R5 R016301 1–3 800 81 † 0.79
R016302 1–3 1,000 82 † 0.78
R016303 1–3 900 82 † 0.76
R016305 1–3 2,400 79 † 0.79
R016306 1–3 800 85 † 0.87
R016307 1–3 600 79 † 0.77
R016308 1–4 800 83 † 0.87
R6 R013201 1–4 2,500 81 † 0.88
R013203 1–2 1,800 94 0.53 †
R013205 1–2 600 98 0.80 †
R013207 1–2 900 90 0.71 †
R013209 1–2 1,400 96 0.91 †
R013211 1–2 3,800 81 0.63 †
R013212 1–4 2,900 81 † 0.77
R7 R013701 1–2 2,000 82 0.64 †
R013702 1–2 1,400 82 0.65 †
R013704 1–2 1,400 86 0.56 †
R013706 1–4 1,000 77 † 0.75
R013708 1–2 1,400 85 0.70 †
R013710 1–2 1,900 90 0.71 †
R013712 1–2 900 81 0.62 †
R8 R016401 1–3 800 82 † 0.77
R016402 1–3 1,400 62 † 0.61
R016403 1–3 1,500 78 † 0.76
R016404 1–3 1,500 86 † 0.71
R016405 1–3 1,100 88 † 0.80
R016407 1–3 1,000 80 † 0.83
R016408 1–4 1,400 82 † 0.84
R9 R016101 1–3 900 90 † 0.87
R016104 1–3 2,400 82 † 0.58
R016107 1–3 1,900 93 † 0.90
R016108 1–3 2,900 75 † 0.74
R016109 1–3 1,500 89 † 0.86
R10 R013402 1–2 1,100 98 0.96 †
R013403 1–4 1,500 96 † 0.97
R013405 1–2 1,400 93 0.81 †
R013406 1–4 2,000 82 † 0.93
R013407 1–2 1,700 92 0.79 †
R013409 1–2 800 89 0.66 †
R013411 1–2 1,600 94 0.82 †
R013412 1–2 1,800 86 0.59 †
R13 R015503 1–2 1,600 93 0.57 †
R015505 1–2 1,400 88 0.76 †
R015507 1–4 1,000 82 † 0.85
R015509 1–2 1,500 86 0.70 †
R015512 1–2 1,100 91 0.80 †
R015514 1–4 600 85 † 0.85
R14 R016501 1–3 900 82 † 0.70
R016502 1–3 1,100 84 † 0.77
R016601 1–3 1,900 78 † 0.72
R016602 1–3 1,400 85 † 0.84
R016603 1–3 3,000 70 † 0.58
R016604 1–3 2,700 79 † 0.75
R016605 1–3 1,300 79 † 0.69
R016701 1–4 1,500 76 † 0.70
†The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment.

Last updated 25 March 2009 (GF)

Printer-friendly Version