Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 8 Science Assessment: 2000
 K036401
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 8 science assessment: 2000
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
S3 K040601 1–3 600 98 † 0.97
K040603 1–3 600 94 † 0.95
K040604 1–3 600 85 † 0.80
K040605 1–3 600 89 † 0.87
K040606 1–4 600 93 † 0.94
K040607 1–4 600 91 † 0.95
K040608 1–4 600 84 † 0.93
K040609 1–4 600 94 † 0.95
K040610 1–4 600 90 † 0.90
S5 K040801 1–3 800 97 † 0.94
K040802 1–3 800 98 † 0.94
K040803 1–3 700 97 † 0.96
K040804 1–3 500 89 † 0.68
K040805 1–3 500 93 † 0.87
K040806 1–2 400 87 0.58 †
K040808 1–3 700 95 † 0.96
K040809 1–3 600 92 † 0.91
S6 K031301 1–4 400 98 † 0.83
K031302 1–3 400 89 † 0.79
K031305 1–3 400 92 † 0.76
K031306 1–3 400 97 † 0.74
K031307 1–3 400 92 † 0.87
K031308 1–3 600 98 † 0.94
K031309 1–4 400 98 † 0.92
S8 K041306 1–3 600 81 † 0.85
K041307 1–3 600 85 † 0.79
K041401 1–3 600 93 † 0.92
K041402 1–3 600 95 † 0.91
K041403 1–3 600 89 † 0.77
S9 K031602 1–3 600 97 † 0.97
K031603 1–3 600 97 † 0.96
K031604 1–3 600 98 † 0.97
K031606 1–3 600 88 † 0.84
K031607 1–4 600 87 † 0.90
K031608 1–3 600 94 † 0.91
K031609 1–3 600 96 † 0.97
K031610 1–3 600 96 † 0.95
K031611 1–3 600 94 † 0.85
K031613 1–3 600 98 † 0.91
S11 K043001 1–3 600 92 † 0.87
K043101 1–3 600 87 † 0.88
K043102 1–4 600 76 † 0.83
K043103 1–3 600 90 † 0.73
K043501 1–3 600 93 † 0.92
K043601 1–3 600 91 † 0.88
K043602 1–3 600 96 † 0.86
K043603 1–3 600 92 † 0.80
S12 K047201 1–4 900 85 † 0.92
K047301 1–3 800 95 † 0.95
K047401 1–3 900 92 † 0.92
K047901 1–3 800 91 † 0.90
K048001 1–3 800 91 † 0.76
K048101 1–4 700 84 † 0.76
K048102 1–3 600 92 † 0.51
K048103 1–2 700 87 0.72 †
S13 K048601 1–3 600 96 † 0.96
K048901 1–3 600 97 † 0.98
K049001 1–3 500 99 † 0.77
K049301 1–3 500 97 † 0.83
K049401 1–3 600 91 † 0.86
K049402 1–3 500 84 † 0.86
K049403 1–3 500 85 † 0.85
K049404 1–4 500 76 † 0.74
S14 K035601 1–3 600 78 † 0.69
K035801 1–3 600 93 † 0.94
K035901 1–3 600 89 † 0.88
K036101 1–3 600 89 † 0.83
K036301 1–3 600 98 † 0.98
K036401 1–3 600 79 † 0.18
K036402 1–3 600 91 † 0.88
K036403 1–3 600 66 † 0.09
K036404 1–3 600 80 † 0.22
K036701 1–3 600 96 † 0.89
K036801 1–3 600 96 † 0.94
S15 K037301 1–3 600 94 † 0.95
K037401 1–3 600 90 † 0.77
K037501 1–3 600 97 † 0.85
K037601 1–3 600 82 † 0.81
K037701 1–4 600 99 † 0.99
K037703 1–3 600 94 † 0.92
K038101 1–2 600 97 0.83 †
K038201 1–3 600 95 † 0.87
K038301 1–5 400 87 † 0.90
S21 K045301 1–3 600 92 † 0.92
K045601 1–4 600 97 † 0.97
K045701 1–3 600 87 † 0.88
K045801 1–3 600 92 † 0.87
K046301 1–3 600 91 † 0.84
K046401 1–3 600 94 † 0.82
K046501 1–3 600 95 † 0.94
K046601 1–3 600 85 † 0.87
K046701 1–4 600 85 † 0.84
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.
Last updated 11 July 2008 (KL)

Printer-friendly Version