NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Writing National Main Assessment: 2002

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Writing National Main Assessment: 2002

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 writing national main assessment: 2002
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
W3	W008302	1–6	1,500	51	†	0.61
W3	W008402	1–6	1,500	70	†	0.86
W6	W008602	1–6	3,000	66	†	0.87
W7	W008702	1–6	1,200	74	†	0.93
W9	W008902	1–6	2,100	71	†	0.88
W11	W009102	1–6	2,000	72	†	0.92
W12	W009202	1–6	1,600	60	†	0.87
W13	W009302	1–6	2,000	68	†	0.93
W14	W009402	1–6	3,200	64	†	0.89
W15	W009502	1–6	3,300	70	†	0.89
W17	W009702	1–6	3,100	69	†	0.91
W18	W009802	1–6	1,900	72	†	0.94
W19	W009902	1–6	2,300	74	†	0.94
W20	W010002	1–6	1,400	65	†	0.89
W22	W010202	1–6	3,100	72	†	0.92
W23	W010302	1–6	1,500	71	†	0.93
W24	W010402	1–6	2,600	67	†	0.91
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Writing Assessment.

Last updated 26 March 2009 (GF)