Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Science Assessment: 2000
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 science assessment: 2000
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
S3 K040601 1?3 1,100 99 ? 1.00
K040603 1?3 1,100 97 ? 0.98
K040604 1?3 1,100 95 ? 0.97
K040605 1?3 1,100 96 ? 0.97
K040606 1?4 1,100 97 ? 0.99
K040607 1?4 1,100 96 ? 0.98
K040608 1?4 1,100 95 ? 0.98
K040609 1?4 1,100 97 ? 0.99
K040610 1?4 1,100 96 ? 0.98
S5 K040801 1?3 1,100 97 ? 0.98
K040802 1?3 1,100 98 ? 0.97
K040803 1?3 1,100 97 ? 0.96
K040804 1?3 1,100 94 ? 0.94
K040805 1?3 1,100 94 ? 0.92
K040806 1?2 1,100 94 0.90 ?
K040808 1?3 1,100 98 ? 0.99
K040809 1?3 1,100 96 ? 0.97
S6 K031301 1?4 1,100 95 ? 0.89
K031302 1?3 1,100 91 ? 0.85
K031305 1?3 1,100 94 ? 0.86
K031306 1?3 1,100 97 ? 0.85
K031307 1?3 1,100 93 ? 0.94
K031308 1?3 1,100 99 ? 0.99
K031309 1?4 1,100 96 ? 0.94
S7 K097901 1?3 1,500 98 ? 0.99
K098501 1?3 1,500 97 ? 0.97
S8 K041306 1?3 1,500 92 ? 0.94
K041307 1?3 1,500 94 ? 0.94
K041401 1?3 1,500 97 ? 0.97
K041402 1?3 1,500 99 ? 0.98
K041403 1?3 1,500 91 ? 0.87
S9 K031602 1?3 1,500 97 ? 0.98
K031603 1?3 1,500 99 ? 0.99
K031604 1?3 1,500 99 ? 0.99
K031606 1?3 1,500 96 ? 0.96
K031607 1?4 1,500 93 ? 0.95
K031608 1?3 1,500 95 ? 0.94
K031609 1?3 1,500 95 ? 0.96
K031610 1?3 1,500 97 ? 0.96
K031611 1?3 1,500 99 ? 0.98
K031613 1?3 1,500 98 ? 0.95
S10 K092601 1?3 1,100 97 ? 0.98
K093601 1?3 1,100 99 ? 0.99
K095801 1?2 1,100 93 0.86 ?
K095901 1?3 1,100 95 ? 0.96
K096101 1?3 1,100 96 ? 0.96
K099001 1?3 1,100 98 ? 0.98
S11 K043001 1?3 1,100 97 ? 0.96
K043101 1?3 1,100 94 ? 0.95
K043102 1?4 1,100 95 ? 0.98
K043103 1?3 1,100 95 ? 0.88
K043501 1?3 1,100 96 ? 0.96
K043601 1?3 1,100 96 ? 0.97
K043602 1?3 1,100 98 ? 0.97
K043603 1?3 1,100 95 ? 0.97
S12 K047201 1?4 1,100 96 ? 0.98
K047301 1?3 1,100 99 ? 0.99
K047401 1?3 1,100 96 ? 0.98
K047901 1?3 1,100 99 ? 0.99
K048001 1?3 1,100 100 ? 1.00
K048101 1?4 1,100 95 ? 0.95
K048102 1?3 1,100 98 ? 0.97
K048103 1?2 1,100 97 0.95 ?
S13 K048601 1?3 1,100 98 ? 0.98
K048901 1?3 1,100 99 ? 0.99
K049001 1?3 1,100 99 ? 0.91
K049301 1?3 1,100 100 ? 1.00
K049401 1?3 1,100 97 ? 0.97
K049402 1?3 1,100 95 ? 0.97
K049403 1?3 1,100 94 ? 0.95
K049404 1?4 1,100 89 ? 0.90
S14 K035601 1?3 1,500 91 ? 0.91
K035801 1?3 1,500 96 ? 0.97
K035901 1?3 1,500 96 ? 0.97
K036101 1?3 1,500 96 ? 0.97
K036301 1?3 1,500 95 ? 0.97
K036401 1?3 1,500 98 ? 0.98
K036402 1?3 1,500 96 ? 0.96
K036403 1?3 1,500 96 ? 0.94
K036404 1?3 1,500 95 ? 0.88
K036701 1?3 1,500 99 ? 0.98
K036801 1?3 1,500 96 ? 0.96
S15 K037301 1?3 1,100 97 ? 0.96
K037401 1?3 1,100 92 ? 0.91
K037501 1?3 1,100 95 ? 0.88
K037601 1?3 1,100 95 ? 0.97
K037701 1?4 1,100 99 ? 0.99
K037703 1?3 1,100 97 ? 0.97
K038101 1?2 1,100 98 0.94 ?
K038201 1?3 1,100 95 ? 0.91
K038301 1?5 1,100 91 ? 0.96
S20 K092801 1?3 1,100 99 ? 0.99
K093701 1?3 1,100 96 ? 0.97
K093901 1?3 1,100 94 ? 0.95
K094901 1?3 1,100 96 ? 0.93
K095401 1?3 1,100 95 ? 0.91
K097001 1?3 1,100 94 ? 0.91
S21 K045301 1?3 1,100 97 ? 0.98
K045601 1?4 1,100 97 ? 0.96
K045701 1?3 1,100 94 ? 0.94
K045801 1?3 1,100 96 ? 0.95
K046301 1?3 1,100 98 ? 0.97
K046401 1?3 1,100 94 ? 0.89
K046501 1?3 1,100 98 ? 0.98
K046601 1?3 1,100 95 ? 0.97
K046701 1?4 1,100 92 ? 0.94
? The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.
Last updated 15 July 2008 (KL)

Printer-friendly Version