Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Science Assessment: 2000
NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 science assessment: 2000
Block Item Range of response codes Sample size Percentage exact agreement Cohen's Kappa Intraclass correlation
S3 K031001 1?3 1,100 96 ? 0.94
K031002 1?3 1,100 91 ? 0.86
K031003 1?3 1,100 88 ? 0.78
K031004 1?2 1,100 91 0.84 ?
K031005 1?3 1,100 89 ? 0.87
K031006 1?3 1,100 94 ? 0.97
K031007 1?3 1,100 90 ? 0.94
S4 K031101 1?2 1,100 98 0.95 ?
K031102 1?2 1,100 97 0.94 ?
K031103 2-2 1,100 96 0.93 ?
K031104 3-2 1,100 99 0.97 ?
K031105 1?3 1,100 100 ? 1.00
K031107 1?4 1,100 96 ? 0.98
S6 K031301 1?4 1,200 94 ? 0.97
K031302 1?3 1,200 95 ? 0.93
K031303 1?3 1,200 94 ? 0.98
K031304 1?3 1,200 97 ? 0.99
K031309 1?4 1,200 92 ? 0.97
S7 K031401 1?4 1,500 89 ? 0.89
K031402 1?3 1,500 93 ? 0.95
K031403 1?3 1,500 94 ? 0.95
K031404 1?3 1,500 97 ? 0.99
K031407 1?3 1,500 94 ? 0.94
K031408 1?3 1,500 98 ? 0.99
K031409 1?3 1,500 98 ? 0.98
K031410 1?3 1,500 97 ? 0.98
S9 K031602 1?3 1,500 98 ? 0.99
K031603 1?3 1,500 99 ? 0.99
K031604 1?3 1,500 99 ? 0.98
K031606 1?3 1,500 96 ? 0.95
K031607 1?4 1,500 95 ? 0.97
K031608 1?3 1,500 96 ? 0.95
K031609 1?3 1,500 99 ? 0.98
S10 K031901 1?3 1,100 95 ? 0.96
K032001 1?3 1,100 97 ? 0.97
K032501 1?3 1,100 93 ? 0.95
K032502 1?3 1,100 92 ? 0.93
K032601 1?3 1,100 96 ? 0.95
K032602 1?3 1,100 96 ? 0.97
S12 K034001 1?3 1,100 97 ? 0.93
K034101 1?3 1,100 96 ? 0.96
K034401 1?3 1,100 95 ? 0.98
K034501 1?4 1,100 93 ? 0.97
K034502 1?3 1,100 99 ? 0.99
S13 K034802 1?3 1,200 93 ? 0.90
K034901 1?3 1,200 93 ? 0.92
K034902 1?3 1,200 93 ? 0.94
K035201 1?3 1,200 94 ? 0.96
K035301 1?4 1,200 95 ? 0.97
S14 K035601 1?3 1,500 92 ? 0.89
K035801 1?3 1,500 93 ? 0.95
K035901 1?3 1,500 98 ? 0.96
K036101 1?3 1,500 96 ? 0.97
K036301 1?3 1,500 97 ? 0.97
S15 K037301 1?3 1,100 98 ? 0.96
K037401 1?3 1,100 96 ? 0.88
K037501 1?3 1,200 99 ? 0.97
K037601 1?3 1,100 96 ? 0.96
K037701 1?4 1,100 98 ? 0.97
K037702 1?3 1,100 98 ? 0.99
S21 K039801 1?3 1,200 97 ? 0.95
K039901 1?5 1,200 93 ? 0.97
K040001 1?3 1,200 93 ? 0.96
K040301 1?3 1,200 93 ? 0.94
K040401 1?4 1,200 95 ? 0.93
K040501 1?3 1,200 99 ? 0.99
S11 K092201 1?3 1,100 95 ? 0.96
S20 K096901 1?3 1,200 94 ? 0.92
S11 K098201 1?3 1,200 95 ? 0.94
K098301 1?3 1,100 94 ? 0.91
S20 K098401 1?3 1,200 97 ? 0.96
S11 K099501 1?3 1,100 97 ? 0.97
S20 K099601 1?3 1,200 92 ? 0.95
? The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.
Last updated 15 July 2008 (KL)

Printer-friendly Version