Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 8 Geography Assessment: 2001
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 8 geography assessment: 2001
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
G3 G013402 1–3 600 86 † 0.77
G014001 1–3 600 98 † 0.97
G014201 1–4 600 88 † 0.94
G014301 1–3 600 86 † 0.88
G014401 1–3 600 95 † 0.92
G5 G016201 1–3 600 97 † 0.97
G016302 1–3 600 97 † 0.93
G016401 1–3 600 95 † 0.96
G016502 1–3 600 92 † 0.86
G016701 1–3 600 91 † 0.92
G017101 1–4 600 79 † 0.87
G8 G012201 1–3 600 98 † 0.98
G012503 1–3 600 100 † 1.00
G012902 1–3 600 98 † 0.98
G013201 1–3 600 79 † 0.82
G9 G019003 1–3 600 95 † 0.94
G019102 1–3 600 97 † 0.95
G019202 1–3 600 91 † 0.88
G019302 1–3 600 94 † 0.94
G019402 1–3 600 94 † 0.95
G019901 1–4 600 93 † 0.94
G020001 1–3 600 97 † 0.89
G020201 1–3 600 94 † 0.90
G020302 1–3 600 92 † 0.90
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.
Last updated 11 July 2008 (KL)

Printer-friendly Version


Back to Top