NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Intraclass Correlation for the Constructed-Response Items From 1998, Rescored in 2002, by Block and Item, Grade 4 Writing Assessment: 2002

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2002, by Block and Item, Grade 4 Writing Assessment: 2002

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and intraclass correlation for the constructed-response items from previous years, rescored in 2002, by block and item, grade 4 writing assessment: 2002
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
W3	W004002	1–6	3,200	71	†	0.92
W4	W004102	1–6	2,800	81	†	0.95
W5	W004202	1–6	4,000	69	†	0.92
W7	W004402	1–6	3,700	79	†	0.95
W8	W004502	1–6	2,800	79	†	0.95
W9	W004602	1–6	3,300	74	†	0.94
W10	W004702	1–6	3,700	75	†	0.94
W11	W004802	1–6	3,000	73	†	0.94
W12	W004902	1–6	2,400	82	†	0.96
W13	W005002	1–6	3,800	70	†	0.92
W14	W005102	1–6	3,400	74	†	0.92
W15	W005202	1–6	2,800	71	†	0.93
W16	W005302	1–6	3,600	72	†	0.91
W19	W005602	1–6	3,700	82	†	0.96
W20	W005702	1–6	5,200	73	†	0.94
W21	W005802	1–6	3,600	76	†	0.94
W22	W005902	1–6	3,500	71	†	0.91
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Writing Assessment.

Last updated 26 March 2009 (GF)