Skip to main content

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 4 Geography Assessment: 2001
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 geography assessment: 2001
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa1 Intraclass correlation2
G4 G008503 1–3 600 94 † 0.93
G008701 1–3 600 94 † 0.94
G009001 1–4 600 83 † 0.89
G5 G009201 1–3 600 88 † 0.87
G009402 1–3 600 94 † 0.91
G009403 1–3 600 94 † 0.88
G009601 1–4 600 91 † 0.96
G8 G012201 1–3 600 97 † 0.97
G012503 1–3 600 99 † 0.99
G012902 1–3 600 98 † 0.99
G013001 1–4 600 94 † 0.97
G013201 1–3 600 91 † 0.82
† Not applicable. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
1 Cohen’s Kappa is a measure of reliability that is appropriate for items that are dichotomously scored.
2 The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.

Last updated 23 January 2009 (GF)

Printer-friendly Version