Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 4 Science Assessment: 2000
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 4 science assessment: 2000
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
S3 K031001 1–3 300 95 † 0.89
K031002 1–3 300 83 † 0.78
K031003 1–3 300 84 † 0.73
K031004 1–2 300 93 0.81 †
K031005 1–3 300 82 † 0.76
K031006 1–3 300 91 † 0.92
K031007 1–3 300 84 † 0.86
S4 K031101 1–2 400 97 0.90 †
K031102 1–2 400 94 0.87 †
K031103 1–2 400 90 0.79 †
K031104 1–2 400 98 0.84 †
K031105 1–3 400 99 † 0.99
K031107 1–4 400 90 † 0.94
S6 K031301 1–4 200 91 † 0.72
K031302 1–3 200 96 † 0.80
K031303 1–3 200 92 † 0.92
K031304 1–3 200 92 † 0.92
K031309 1–4 200 94 † 0.94
S7 K031401 1–4 600 78 † 0.54
K031402 1–3 600 86 † 0.59
K031403 1–3 600 84 † 0.57
K031404 1–3 400 83 † 0.53
K031407 1–3 600 75 † 0.46
K031408 1–3 600 87 † 0.56
K031409 1–3 600 85 † 0.63
K031410 1–3 600 86 † 0.62
S9 K031602 1–3 600 89 † 0.81
K031603 1–3 600 92 † 0.64
K031604 1–3 600 92 † 0.82
K031606 1–3 600 80 † 0.48
K031607 1–4 600 78 † 0.70
K031608 1–3 600 83 † 0.55
K031609 1–3 600 87 † 0.65
S10 K031901 1–3 600 77 † 0.63
K032001 1–3 600 87 † 0.74
K032501 1–3 600 83 † 0.60
K032502 1–3 600 87 † 0.53
K032601 1–3 600 84 † 0.68
K032602 1–3 600 84 † 0.64
S12 K034001 1–3 600 85 † 0.62
K034101 1–3 600 83 † 0.72
KW34101 1–3 600 89 † 0.91
KX34101 1–3 600 88 † 0.67
KY34101 1–3 600 75 † 0.67
KZ34101 1–3 600 82 † 0.53
KZ34401 1–3 600 82 † 0.67
KZ34501 1–4 600 81 † 0.64
KZ34502 1–3 600 89 † 0.81
S13 K034802 1–3 600 94 † 0.90
K034901 1–3 600 86 † 0.54
K034902 1–3 600 87 † 0.74
K035201 1–3 600 80 † 0.73
K035301 1–4 600 85 † 0.53
S14 K035601 1–3 600 81 † 0.56
K035801 1–3 600 85 † 0.83
K035901 1–3 600 83 † 0.60
K036101 1–3 600 82 † 0.64
K036301 1–3 600 86 † 0.76
S15 K037301 1–3 600 87 † 0.67
K037401 1–3 600 81 † 0.24
K037501 1–3 600 91 † 0.26
K037601 1–3 600 79 † 0.58
K037701 1–4 600 91 † 0.56
K037702 1–3 600 83 † 0.58
S21 K039801 1–3 600 87 † 0.54
K039901 1–5 400 78 † 0.69
K040001 1–3 600 83 † 0.69
K040301 1–3 600 80 † 0.64
K040401 1–4 600 80 † 0.76
K040501 1–3 600 84 † 0.64
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.
Last updated 15 July 2008 (KL)

Printer-friendly Version