Skip Navigation
small NCES header image

Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Assessment: 2005
Range of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 reading combined national and state main assessment: 2005
Block Item Range of response codes Sample size Percent exact agreement Cohen's Kappa Intraclass correlation
R3 R017101 1–2 1,200 93 0.86
R017102 1–3 1,000 88 0.86
R017104 1–3 1,100 95 0.94
R017105 1–4 1,100 77 0.82
R017107 1–3 900 90 0.89
R017108 1–2 1,000 96 0.83
R017110 1–3 1,000 95 0.95
R4 R018401 1–3 1,200 84 0.84
R018601 1–3 1,200 77 0.76
R018701 1–3 1,100 80 0.81
R018801 1–3 1,100 84 0.85
R019001 1–4 1,100 73 0.77
R019101 1–3 1,100 83 0.81
R019201 1–3 1,000 81 0.81
R5 R012601 1–2 1,200 91 0.82
R012604 1–2 1,100 96 0.92
R012607 1–4 1,100 78 0.83
R012612 1–2 1,100 90 0.79
R6 R013201 1–4 1,200 89 0.93
R013203 1–2 1,200 96 0.72
R013205 1–2 1,100 96 0.80
R013207 1–2 1,100 90 0.76
R013209 1–2 1,100 96 0.88
R013211 1–2 1,000 86 0.72
R013212 1–4 900 82 0.77
R7 R012702 1–2 1,200 94 0.71
R012703 1–2 1,200 85 0.70
R012705 1–2 1,100 87 0.74
R012706 1–2 1,200 85 0.68
R012710 1–2 1,100 93 0.81
R012713 1–2 1,000 98 0.96
R012714 1–4 1,100 75 0.79
R8 R053401 1–3 1,200 85 0.87
R053402 1–2 1,200 91 0.60
R053405 1–4 1,100 76 0.77
R053408 1–3 1,100 87 0.86
R053410 1–2 1,100 91 0.77
R9 R053301 1–3 1,200 93 0.87
R053303 1–2 1,100 90 0.80
R053305 1–3 1,100 86 0.90
R053309 1–3 1,100 88 0.90
R10 R013402 1–2 1,200 99 0.98
R013403 1–4 1,100 96 0.96
R013405 1–2 1,100 95 0.89
R013406 1–4 1,100 81 0.92
R013407 1–2 1,100 96 0.93
R013409 1–2 1,000 93 0.85
R013411 1–2 1,000 91 0.79
R013413 1–2 1,000 84 0.67
R11 R013001 1–2 1,200 95 0.85
R013003 1–2 1,200 100 0.99
R013004 1–4 1,100 83 0.92
R013005 1–2 1,100 91 0.75
R013007 1–2 1,100 97 0.89
R013008 1–2 1,100 87 0.75
R013009 1–2 1,000 91 0.67
R013010 1–2 1,000 97 0.90
R013011 1–2 1,000 90 0.77
R12 R053202 1–3 1,200 88 0.86
R053203 1–3 1,100 86 0.84
R053205 1–4 1,100 77 0.87
R053207 1–3 1,000 84 0.81
R053209 1–3 1,100 89 0.87
R13 R016201 1–3 800 92 0.67
R016202 1–3 800 90 0.80
R016204 1–4 800 91 0.83
R016205 1–3 800 82 0.70
R016207 1–3 700 95 0.96
R016210 1–4 700 76 0.76
R016211 1–3 800 86 0.76
R016212 1–3 800 88 0.90
R016213 1–3 800 86 0.74
R14 R026501 1–3 1,200 84 0.83
R026801 1–4 1,100 78 0.84
R027001 1–2 1,000 89 0.69
R027201 1–3 1,100 87 0.84
R15 R028401 1–3 1,200 94 0.92
R028501 1–3 1,200 93 0.91
R028801 1–4 1,100 85 0.89
R029601 1–3 1,100 87 0.80
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items.
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version


Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education