NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years Rescored in 2006, Grade 4 U.S. History Assessment, by Item and Block: 2006

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years Rescored in 2006, Grade 4 U.S. History Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years rescored in 2006, grade 4 U.S. history assessment, by item and block: 2006
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
H4	H054301	1–3	700	82.78	†	0.85
	H054601	1–3	700	88.27	†	0.85
	H054801	1–3	700	88.92	†	0.88
	H054901	1–3	700	88.03	†	0.89
	H055301	1–3	700	86.68	†	0.85
H5	H055901	1–3	800	89.75	†	0.90
	H056101	1–3	800	89.24	†	0.90
	H056401	1–3	800	93.62	†	0.92
	H056601	1–3	700	87.92	†	0.86
	H056801	1–3	800	96.82	†	0.95
H7	H057501	1–3	800	96.11	†	0.94
	H057701	1–3	800	82.91	†	0.79
	H057801	1–3	700	82.93	†	0.76
	H058601	1–3	800	82.13	†	0.82
	H058701	1–3	800	83.93	†	0.82
H8	H034101	1–4	700	93.25	†	0.80
	H034401	1–3	700	90.95	†	0.90
	H034501	1–2	700	99.86	0.99	†
	H034702	1–3	700	98.49	†	0.93
	H035001	1–3	700	85.04	†	0.84
	H035101	1–3	700	71.62	†	0.75
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 U.S. History Assessment.

Last updated 19 November 2009 (GF)