Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2003, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2003
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous assessment years, rescored in 2003, by block and item, grade 4 reading combined national and state main assessment: 2003
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
R3 R017001 1–2 1,400 88 0.74 †
R017003 1–3 2,000 86 † 0.86
R017004 1–2 1,400 94 0.85 †
R017006 1–2 1,600 92 0.78 †
R017007 1–4 2,200 73 † 0.86
R017009 1–3 3,000 85 † 0.78
R4 R012102 1–2 600 94 0.88 †
R012104 1–2 1000 93 0.86 †
R012106 1–2 1,000 86 0.72 †
R012109 1–2 200 89 0.74 †
R012111 1–4 3,000 89 † 0.92
R012112 1–2 1,000 87 0.73 †
R012113 1–3 800 88 † 0.87
R5 R012601 1–2 1000 85 0.68 †
R012604 1–2 2,000 93 0.84 †
R012607 1–4 3,600 80 † 0.83
R012612 1–2 3,000 83 0.67 †
R6 R017301 1–2 2,400 90 0.80 †
R017303 1–3 2,000 86 † 0.88
R017307 1–4 4,200 77 † 0.86
R017309 1–3 3,200 83 † 0.84
R017310 1–3 2,000 85 † 0.84
R7 R012702 1–2 1,000 91 0.79 †
R012703 1–2 800 87 0.74 †
R012705 1–2 2,000 91 0.78 †
R012706 1–2 1,200 80 0.56 †
R012710 1–2 600 89 0.76 †
R012714 1–4 3,800 80 † 0.82
R8 R017401 1–3 2,000 83 † 0.85
R017701 1–3 1,200 85 † 0.87
R017901 1–4 4,400 79 † 0.89
R018201 1–3 3,400 85 † 0.87
R018301 1–3 2,800 84 † 0.85
R9 R015803 1–3 1,800 85 † 0.83
R015804 1–4 3,200 78 † 0.81
R015806 1–3 2,600 81 † 0.81
R015807 1–3 1,400 82 † 0.81
R015809 1–3 1,800 87 † 0.87
R015810 1–3 2,200 88 † 0.87
R10 R012503 1–2 600 87 0.75 †
R012504 1–2 600 97 0.93 †
R012506 1–2 600 89 0.78 †
R012508 1–2 600 97 0.94 †
R012511 1–2 600 93 0.85 †
R012512 1–4 2,800 76 † 0.86
R11 R020401 1–3 3,200 86 † 0.88
R020701 1–2 800 88 0.77 †
R021201 1–3 2,400 84 † 0.86
R021701 1–3 2,200 79 † 0.81
R12 R023201 1–2 1,200 91 0.82 †
R023501 1–3 2,200 85 † 0.89
R023601 1–3 1,200 84 † 0.87
R024001 1–3 3,600 80 † 0.82
†The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Reading Assessment.
Last updated 25 March 2009 (GF)

Printer-friendly Version


Back to Top