Skip to main content
Skip Navigation
IES    
NCES
National Center for
Education Statistics
Menu
Contact
Tools
ED.gov
Newsflash
Search IES

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percent Agreement for the Constructed-Response Items From the Previous Year 2011, Rescored in 2013, Grade 8 Mathematics National and State Assessment, by Item and Block: 2013
Score range and percent agreement for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessment, by item and block: 2013
BlockItemRange of response codesSample sizePercent agreement
M3M1499011-31,70098
M3M1508011-31,60097
M3M1509011-21,70099
M3M1512011-21,70099
M4M1515011-31,70092
M4M1516011-31,70095
M4M1526021-51,70097
M5M2212031-31,70099
M5M2212041-31,70086
M5M2217011-21,70099
M5M2223011-31,70099
M6M1413511-31,70095
M6M1416011-21,70092
M6M1419011-51,70088
M7M1653011-31,60094
M7M1670011-21,70092
M7M1685011-21,60099
M7M1685021-21,60099
M7M1685031-31,60092
M7M1687011-31,60097
M7M1701011-31,60099
M10M2227011-31,70096
M10M2231011-31,70099
M10M2233011-31,70093
M10M2238011-31,70098
M11M1049011-31,70097
M11M1629011-31,70096
M11M1631011-31,70095
M11M1679011-31,70094
M11M1683011-31,70096
NOTE: Blocks M4, M5, M9, M10, M11, and M12 have larger sample sizes due to their pairing with integrated pilot blocks. Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment.

Score range and Cohen's kappa or intraclass correlation for the constructed-response items from the previous year assessed that were rescored in 2011, by block and item, grade 8 math assessment: 2013
BlockItemRange of response codesSample sizeCohen’s KappaIntraclass correlation
M3M1499011-31,500.98.99
M3M1508011-31,500.93.97
M3M1509011-21,500.99†
M3M1512011-21,500.99†
M4M1515011-31,600.88.93
M4M1516011-31,500.85.93
M4M1526021-51,500.96.98
M5M2212031-31,500.99.99
M5M2212041-31,500.72.79
M5M2217011-21,500.99†
M5M2223011-31,500.98.99
M6M1413511-31,500.91.97
M6M1416011-21,500.81†
M6M1419011-51,500.83.95
M7M1653011-31,500.90.96
M7M1670011-21,500.71†
M7M1685011-21,600.98†
M7M1685021-21,500.97†
M7M1685031-31,500.78.89
M7M1687011-31,500.95.98
M7M1701011-31,500.98.99
M10M2227011-31,500.93.97
M10M2231011-31,500.98.99
M10M2233011-31,500.88.91
M10M2238011-31,500.97.99
M11M1049011-31,500.95.97
M11M1629011-31,500.94.96
M11M1631011-31,500.93.96
M11M1679011-31,500.85.93
M11M1683011-31,500.93.97
† The Intraclass correlation is not reported for dichotomously scored items.
NOTE: Blocks M4, M5, M9, M10, M11, and M12 have larger sample sizes due to their pairing with integrated pilot blocks. The intraclass correlation is most appropriate for items with more than two categories. Cohen"s Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessment.
Last updated 13 June 2017 (GF)
IES    
NCES
National Center for
Education Statistics
Explore the Institute of Education Sciences