NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Age 13 Mathematics Long-Term Trend Bridge Study: 2004
Range of response codes, percent exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, age 13 mathematics long-term trend bridge study: 2004
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa
M1 N257601 1 - 2 600 99 0.98
N263101 1 - 2 600 99 0.98
N275001 1 - 2 600 98 0.95
N276801 1 - 2 700 100 0.89
N276802 1 - 2 700 100 0.97
N276803 1 - 2 600 99 0.97
N277601 1 - 2 600 100 1.00
N277602 1 - 2 600 99 0.98
N277603 1 - 2 600 99 0.96
M2 N256101 1 - 2 700 98 0.87
N269201 1 - 2 700 97 0.94
N277901 1 - 2 800 100 0.97
N277902 1 - 2 800 99 0.95
N277903 1 - 2 800 99 0.97
N286601 1 - 2 700 98 0.96
N286602 1 - 2 700 98 0.96
N286603 1 - 2 700 99 0.98
M21 N292401 1 - 2 1,400 100 0.99
N300201 1 - 2 1,300 99 0.98
N300401 1 - 2 1,300 100 1.00
N300501 1 - 2 1,400 100 0.99
N301501 1 - 2 1,400 100 0.99
M22 N301801 1 - 2 800 100 0.96
N302401 1 - 2 700 99 0.98
N302601 1 - 2 700 100 1.00
N302901 1 - 2 700 100 0.98
N303201 1 - 2 700 100 1.00
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment.

Last updated 06 April 2009 (GF)