Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Science Combined National and State Main Assessment: 2005
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 science combined national and state main assessment: 2005
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
B3 K040601 1–3 900 98 † 0.97
K040603 1–3 900 94 † 0.93
K040604 1–3 900 88 † 0.85
K040605 1–3 800 89 † 0.89
K040606 1–4 800 90 † 0.91
K040607 1–4 1,000 91 † 0.95
K040608 1–4 1,000 89 † 0.95
K040609 1–4 1,000 96 † 0.98
K040610 1–4 1,000 93 † 0.96
B5 K040801 1–3 1,000 97 † 0.95
K040802 1–3 1,000 99 † 0.93
K040803 1–3 800 96 † 0.95
K040804 1–3 700 93 † 0.91
K040805 1–3 600 94 † 0.90
K040806 1–2 600 87 0.60 †
K040808 1–3 800 97 † 0.98
K040809 1–3 800 93 † 0.95
B6 K031301 1–4 1,100 97 † 0.82
K031302 1–3 1,100 87 † 0.73
K031305 1–3 1,000 92 † 0.74
K031306 1–2 1,000 97 0.70 †
K031307 1–3 900 92 † 0.85
K031308 1–3 900 95 † 0.72
K031309 1–4 1,100 97 † 0.89
B7 K097901 1–3 1,200 92 † 0.87
K098501 1–3 1,400 93 † 0.93
K101201 1–3 1,200 86 † 0.59
K102001 1–4 1,300 81 † 0.82
K102301 1–3 1,400 96 † 0.96
B8 K041306 1–3 1,500 85 † 0.88
K041307 1–3 1,500 85 † 0.80
K041401 1–3 1,400 88 † 0.86
K041402 1–3 1,400 93 † 0.87
K041403 1–3 1,400 91 † 0.84
B10 K092601 1–3 900 89 † 0.84
K095801 1–2 900 92 0.78 †
K095901 1–3 1,000 94 † 0.92
K096101 1–3 1,000 92 † 0.87
K099001 1–3 1,000 92 † 0.88
B12 K047201 1–4 1,000 87 † 0.93
K047301 1–3 900 91 † 0.89
K047401 1–3 1,000 94 † 0.93
K047901 1–3 900 88 † 0.90
K048001 1–3 1,000 95 † 0.89
B13 K048601 1–3 1,000 95 † 0.94
K048901 1–3 1,000 95 † 0.96
K049301 1–3 1,000 99 † 0.96
K049401 1–3 900 90 † 0.81
K049402 1–3 900 84 † 0.85
K049403 1–3 900 85 † 0.76
K049404 1–4 900 78 † 0.67
B14 K035601 1–3 900 90 † 0.86
K035801 1–3 1,000 91 † 0.93
K035901 1–3 900 92 † 0.90
K036101 1–3 900 87 † 0.84
K036301 1–3 900 93 † 0.94
K036401 1–3 1,000 94 † 0.91
K036402 1–3 900 88 † 0.84
K036403 1–3 900 90 † 0.82
K036404 1–3 900 89 † 0.70
K036801 1–3 800 94 † 0.93
B15 K037301 1–3 900 95 † 0.95
K037401 1–3 700 89 † 0.80
K037601 1–3 900 86 † 0.86
K037701 1–4 1,000 99 † 0.98
K037703 1–3 900 88 † 0.85
K038101 1–2 900 97 0.87 †
K038201 1–3 900 91 † 0.74
K038301 1–5 800 86 † 0.90
B20 K092801 1–3 1,000 96 † 0.94
K093701 1–3 800 95 † 0.93
K093901 1–3 900 88 † 0.88
K094901 1–3 900 91 † 0.78
K095401 1–3 900 94 † 0.84
K097001 1–3 900 89 † 0.82
S9 K060301 1–3 1,400 94 † 0.92
K063403 1–3 1,400 87 † 0.81
K063404 1–3 1,500 93 † 0.91
K063405 1–4 1,500 99 † 0.99
K063406 1–3 1,500 85 † 0.84
K063407 1–3 1,500 90 † 0.91
K063410 1–4 1,500 86 † 0.91
S15 K063901 1–3 1,000 98 † 0.97
K064101 1–4 900 97 † 0.96
K064501 1–4 900 82 † 0.87
K064801 1–3 900 93 † 0.89
K064802 1–3 900 93 † 0.93
K064803 1–3 900 94 † 0.94
S16 K065601 1–4 900 93 † 0.96
K065901 1–4 900 84 † 0.85
K065902 1–3 900 93 † 0.92
K066101 1–3 900 96 † 0.96
S17 K066701 1–3 900 88 † 0.86
K066901 1–2 900 99 0.95 †
K067201 1–3 1,000 94 † 0.94
K067401 1–3 900 89 † 0.90
K067701 1–4 900 96 † 0.98
K067901 1–3 900 92 † 0.92
S18 K063001 1–3 800 90 † 0.92
K068801 1–3 900 82 † 0.82
K069101 1–3 1,000 94 † 0.93
K069501 1–3 900 90 † 0.76
K069801 1–4 900 90 † 0.88
K069901 1–4 900 96 † 0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.
Last updated 06 April 2009 (GF)

Printer-friendly Version