NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Writing National Main Assessment, by Item and Block: 2007

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Writing National Main Assessment, by Item and Block: 2007

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 writing national main assessment, by item and block: 2007
Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2007 Writing Assessment.
W3	W011802	1–6	700	60.97	—	0.79
W4	W008402	1–6	700	63.68	—	0.71
W5	W011202	1–6	700	72.97	—	0.81
W6	W008602	1–6	700	67.55	—	0.81
W7	W008702	1–6	700	68.60	—	0.78
W9	W008902	1–6	700	67.63	—	0.80
W10	W011302	1–6	700	76.66	—	0.86
W11	W009102	1–6	700	79.32	—	0.86
W12	W009202	1–6	700	68.35	—	0.85
W13	W009302	1–6	700	58.17	—	0.77
W14	W011902	1–6	700	61.77	—	0.79
W15	W009502	1–6	700	59.41	—	0.82
W17	W009702	1–6	700	66.67	—	0.86
W18	W009802	1–6	700	57.23	—	0.77
W19	W009902	1–6	700	65.18	—	0.80
W20	W012002	1–6	700	64.45	—	0.82
W21	W011402	1–6	700	56.34	—	0.73
W22	W010202	1–6	700	61.27	—	0.80
W23	W010302	1–6	700	64.63	—	0.83
W24	W010402	1–6	700	61.73	—	0.82

Last updated 02 March 2010 (JL)