Page Title:
Keywords:
Description:
Skip to main content
Skip Navigation
IES    
NCES
National Center for
Education Statistics
Menu
Contact
Tools
ED.gov
Newsflash
Search IES
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 Reading Combined National and State Assessments, by Item and Block: 2011
NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 4 reading combined national and state assessments, by item and block: 2011
Block Item Range of response codes Sample size Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
 
R1 R057704 1-2 2,100 88
R057705 1-3 2,100 85
R057706 1-3 2,100 85
R057707 1-4 2,100 82
R057710 1-3 2,100 85
R2 R057806 1-4 2,100 85
R057807 1-2 2,100 95
R057809 1-3 2,100 91
R3 R068704 1-3 2,100 82
R068707 1-4 2,100 82
R068708 1-3 2,100 84
R068711 1-3 2,100 93
R4 R058204 1-3 2,100 88
R058206 1-4 2,000 85
R058207 1-3 2,000 91
R058209 1-2 2,000 96
R5 R058504 1-3 2,100 79
R058508 1-4 2,100 76
R058509 1-2 2,100 96
R6 R058602 1-3 2,100 76
R058606 1-3 2,100 88
R058608 1-4 2,100 81
R7 R058805 1-3 2,100 84
R058807 1-4 2,100 77
R058810 1-3 2,100 85
R8 R059106 1-4 2,100 77
R059108 1-3 2,100 86
R059109 1-3 2,100 82
R059110 1-2 2,100 93
R9 R058302 1-2 2,100 97
R058304 1-3 2,000 94
R058307 1-4 2,000 83
R10 R059504 1-3 2,100 79
R059507 1-4 2,100 76
R059509 1-3 2,000 83
R059510 1-3 2,100 91

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling,  grade 4 reading national and state assessments, by item and block: 2011
Block Item Range of response codes Sample size Cohen’s Kappa Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1 R057704 1-2 2,000 0.74 †
R057705 1-3 2,000 0.76 0.85
R057706 1-3 1,900 0.76 0.81
R057707 1-4 1,900 0.73 0.87
R057710 1-3 1,600 0.69 0.78
R2 R057806 1-4 1,900 0.75 0.85
R057807 1-2 1,800 0.84 †
R057809 1-3 1,600 0.80 0.85
R3 R068704 1-3 2,000 0.67 0.79
R068707 1-4 1,900 0.72 0.78
R068708 1-3 1,800 0.72 0.81
R068711 1-3 1,600 0.87 0.90
R4 R058204 1-3 2,000 0.81 0.86
R058206 1-4 1,700 0.71 0.85
R058207 1-3 1,900 0.85 0.91
R058209 1-2 1,600 0.85 †
R5 R058504 1-3 1,900 0.59 0.68
R058508 1-4 1,700 0.61 0.81
R058509 1-2 1,500 0.89 †
R6 R058602 1-3 1,900 0.60 0.70
R058606 1-3 1,800 0.78 0.80
R058608 1-4 1,700 0.63 0.78
R7 R058805 1-3 1,900 0.74 0.83
R058807 1-4 1,800 0.64 0.83
R058810 1-3 1,600 0.69 0.78
R8 R059106 1-4 1,700 0.53 0.70
R059108 1-3 1,600 0.70 0.82
R059109 1-3 2,000 0.69 0.75
R059110 1-2 1,500 0.83 †
R9 R058302 1-2 2,000 0.94 †
R058304 1-3 2,000 0.91 0.95
R058307 1-4 1,800 0.74 0.90
R10 R059504 1-3 1,900 0.63 0.74
R059507 1-4 1,800 0.58 0.75
R059509 1-3 1,600 0.66 0.78
R059510 1-3 1,400 0.79 0.86
Last updated 06 January 2016 (GF)
IES    
NCES
National Center for
Education Statistics
Explore the Institute of Education Sciences