Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Mathematics Assessment: 2000
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 mathematics assessment: 2000
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
M3 M039201 1–2 1,700 100 0.99 †
M039301 1–3 1,700 100 † 1.00
M040001 1–3 1,700 98 † 0.99
M040201 1–2 1,700 95 0.90 †
M5 M066301 1–3 1,700 98 † 0.98
M066501 1–3 1,700 97 † 0.98
M066601 1–3 1,700 96 † 0.98
M066701 1–3 1,700 98 † 0.99
M066801 1–3 1,700 96 † 0.97
M066901 1–5 1,700 93 † 0.98
M6 M019701 1–2 1,700 99 0.99 †
M019801 1–3 1,700 97 † 0.97
M019901 1–3 1,700 99 † 0.99
M020001 1–2 1,700 100 0.99 †
M020101 1–2 1,700 98 0.96 †
M020201 1–2 1,700 98 0.96 †
M020301 1–4 1,700 99 † 1.00
M020401 1–2 1,700 100 1.00 †
M020501 1–2 1,700 99 0.98 †
M020701 1–4 1,700 89 † 0.89
N277903 1–2 1,700 99 0.98 †
M7 M067901 1–3 1,700 98 † 0.98
M068001 1–3 1,700 98 † 0.98
M068002 1–3 1,700 98 † 0.99
M068003 1–3 1,700 99 † 0.98
M068004 1–5 1,700 94 † 0.98
M8 M010631 1–3 1,700 98 † 0.98
M091101 1–3 1,700 98 † 0.99
M091201 1–3 1,700 92 † 0.95
M091401 1–5 1,700 92 † 0.97
M10 M085401 1–5 1,700 95 † 0.97
M085701 1–3 1,700 97 † 0.98
M085901 1–3 1,700 97 † 0.98
M11 M046001 1–5 1,700 99 † 1.00
M046601 1–4 1,700 98 † 0.99
M046801 1–5 1,700 99 † 0.99
M046901 1–5 1,700 99 † 0.99
M047301 1–4 1,700 98 † 0.98
M12 M086601 1–3 1,700 98 † 0.98
M087001 1–3 1,700 98 † 0.99
M087301 1–5 1,700 96 † 0.98
M13 M043201 1–2 1,700 97 0.92 †
M043301 1–3 1,700 97 † 0.96
M043401 1–4 1,700 95 † 0.96
M043402 1–4 1,700 95 † 0.96
M043403 1–3 1,700 98 † 0.97
M043501 1–5 1,700 92 † 0.97
M14 M072201 1–3 1,700 99 † 0.99
M072202 1–3 1,700 97 † 0.97
M072401 1–3 1,700 95 † 0.96
M072501 1–3 1,700 95 † 0.96
M072601 1–3 1,700 98 † 0.98
M072701 1–5 1,700 94 † 0.97
M15 M074301 1–2 1,700 99 0.98 †
M074501 1–3 1,700 95 † 0.96
M074701 1–3 1,700 98 † 0.99
M074801 1–3 1,700 99 † 1.00
M074901 1–3 1,700 97 † 0.98
M075001 1–3 1,700 99 † 0.99
M075101 1–5 1,700 92 † 0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.
Last updated 15 July 2008 (KL)

Printer-friendly Version