Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Writing National Main Assessment, by Item and Block: 2007

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 writing national main assessment, by item and block: 2007
Block Item Range of response codes Sample size Percent agreement Cohen’s Kappa Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2007 Writing Assessment.
W3 W011802 1–6 700 60.97 — 0.79
W4 W008402 1–6 700 63.68 — 0.71
W5 W011202 1–6 700 72.97 — 0.81
W6 W008602 1–6 700 67.55 — 0.81
W7 W008702 1–6 700 68.60 — 0.78
W9 W008902 1–6 700 67.63 — 0.80
W10 W011302 1–6 700 76.66 — 0.86
W11 W009102 1–6 700 79.32 — 0.86
W12 W009202 1–6 700 68.35 — 0.85
W13 W009302 1–6 700 58.17 — 0.77
W14 W011902 1–6 700 61.77 — 0.79
W15 W009502 1–6 700 59.41 — 0.82
W17 W009702 1–6 700 66.67 — 0.86
W18 W009802 1–6 700 57.23 — 0.77
W19 W009902 1–6 700 65.18 — 0.80
W20 W012002 1–6 700 64.45 — 0.82
W21 W011402 1–6 700 56.34 — 0.73
W22 W010202 1–6 700 61.27 — 0.80
W23 W010302 1–6 700 64.63 — 0.83
W24 W010402 1–6 700 61.73 — 0.82

 

Last updated 02 March 2010 (JL)

Printer-friendly Version


Back to Top