Page Title:
Keywords:
Description:
Skip to main content
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 mathematics combined national and state main assessment, by item and block: 2009
Block Item Range of
response codes
Sample size Percent agreement Cohen’s Kappa Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Mathematics Assessment.
M3 M149901 1–3 1,600 98 — 0.98
M150801 1–3 1,500 98 — 0.98
M150901 1–2 1,600 100 0.99 —
M151201 1–2 1,500 100 0.99 —
M4 M151501 1–3 1,500 95 — 0.96
M151601 1–3 1,400 96 — 0.95
M152602 1–5 1,500 99 — 0.99
M5 M019701 1–2 1,600 100 0.98 —
M020001 1–2 1,500 100 0.99 —
M047901 1–3 1,400 100 — 0.99
M053101 1–5 1,200 91 — 0.95
M066301 1–3 1,500 98 — 0.96
M066501 1–3 1,500 97 — 0.97
M067901 1–3 1,400 99 — 0.98
M085701 1–3 1,600 99 — 0.99
M085901 1–3 1,500 98 — 0.98
M6 M141301 1–4 1,500 98 — 0.97
M141601 1–2 1,500 88 0.74 —
M141901 1–5 1,400 90 — 0.94
M7 M165301 1–3 1,500 98 — 0.98
M167001 1–2 1,400 94 0.69 —
M168501 1–2 1,300 99 0.98 —
M168502 1–2 1,300 99 0.98 —
M168503 1–3 1,100 96 — 0.91
M168701 1–3 1,600 98 — 0.99
M170101 1–3 1,500 100 — 1.00
M8 M164601 1–3 1,500 100 — 0.99
M166301 1–3 1,600 96 — 0.97
M169901 1–2 1,500 99 0.97 —
M169902 1–2 1,500 100 0.99 —
M169903 1–3 1,400 97 — 0.97
M169904 1–2 1,300 98 0.92 —
M170501 1–3 1,500 95 — 0.94
M9 M165001 1–2 1,400 99 0.96 —
M165501 1–3 1,600 99 — 0.99
M165701 1–3 1,500 99 — 0.99
M165702 1–3 1,400 96 — 0.97
M168901 1–2 1,600 99 0.98 —
M170801 1–3 1,500 99 — 0.95
M10 M142801 1–2 1,500 100 1.00 —
M143201 1–2 1,500 100 1.00 —
M143501 1–5 1,500 94 — 0.95
M11 M104901 1–3 1,400 99 — 0.98
M162901 1–3 1,600 97 — 0.97
M163101 1–3 1,600 96 — 0.97
M167901 1–3 1,500 98 — 0.97
M168301 1–3 1,400 97 — 0.97
M12 M153201 1–3 1,600 96 — 0.95
M153601 1–2 1,600 100 0.99 —
M153901 1–3 1,500 92 — 0.94

Last updated 06 January 2016 (GF)