Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2002, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2002
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2002, by block and item, grade 4 reading combined national and state main assessment: 2002
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
R3 R017001 1–2 2,800 82 0.62 †
R017003 1–3 5,100 73 † 0.76
R017004 1–2 1,800 89 0.72 †
R017006 1–2 1,200 90 0.72 †
R017007 1–4 3,900 72 † 0.84
R017009 1–3 2,600 85 † 0.78
R4 R012102 1–2 2,400 95 0.90 †
R012104 1–2 1,500 94 0.88 †
R012106 1–2 2,100 89 0.78 †
R012108 1–2 900 94 0.87 †
R012109 1–2 2,800 92 0.83 †
R012111 1–4 3,400 84 † 0.88
R012112 1–2 800 88 0.75 †
R5 R012601 1–2 4,500 86 0.69 †
R012604 1–2 1,800 93 0.83 †
R012607 1–4 2,700 79 † 0.83
R012611 1–2 2,500 87 0.73 †
R6 R017301 1–2 1,800 90 0.80 †
R017303 1–3 1,600 82 † 0.84
R017310 1–3 4,000 87 † 0.88
R017307 1–4 4,600 76 † 0.86
R017309 1–3 3,000 82 † 0.84
R7 R012702 1–2 2,400 89 0.76 †
R012703 1–2 1,700 87 0.73 †
R012705 1–2 2,800 89 0.73 †
R012706 1–2 3,300 82 0.60 †
R012708 1–4 5,400 82 † 0.82
R012710 1–2 2,800 88 0.76 †
R9 R015802 1–2 2,600 87 0.74 †
R015803 1–3 3,900 85 † 0.83
R015804 1–4 5,100 76 † 0.81
R015806 1–3 3,200 82 † 0.81
R015807 1–3 4,200 81 † 0.80
R015809 1–3 3,100 85 † 0.84
R10 R012503 1–2 1,900 87 0.73 †
R012504 1–2 1,300 97 0.94 †
R012506 1–2 2,400 89 0.79 †
R012508 1–2 1,700 97 0.93 †
R012511 1–2 2,500 94 0.86 †
R012512 1–4 3,700 79 † 0.88
†The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment.
Last updated 25 March 2009 (GF)

Printer-friendly Version