NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Civics Assessment, by Item and Block: 2006

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Civics Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 civics assessment, by item and block: 2006
Block	Item	Range of response codes	Sample size	Percent of exact agreement	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 Civics Assessment.
P3	P041503	1–3	400	74	†	0.74
	P041505	1–3	500	75	†	0.72
	P041509	1–4	500	69	†	0.75
	P041511	1–3	500	77	†	0.80
P4	P041606	1–3	500	84	†	0.86
	P041613	1–3	500	97	†	0.97
	P041614	1–3	500	83	†	0.77
P5	P041705	1–3	500	70	†	0.66
	P041706	1–4	500	83	†	0.88
	P041711	1–3	500	91	†	0.88
	P041713	1–3	400	87	†	0.90
P6	P041804	1–3	600	80	†	0.80
	P041806	1–3	500	82	†	0.84
	P041815	1–3	500	80	†	0.78
P7	P041902	1–3	500	84	†	0.85
	P041905	1–4	600	86	†	0.86
	P041907	1–4	400	89	†	0.92
	P041912	1–3	500	77	†	0.72
P8	P042191	1–3	600	89	†	0.90
	P042195	1–4	400	88	†	0.93
	P042199	1–3	500	78	†	0.68
	P042203	1–3	500	85	†	0.85
P9	P042210	1–3	400	87	†	0.90
	P042214	1–4	700	66	†	0.78
	P042218	1–4	500	93	†	0.97
	P042222	1–3	500	86	†	0.86
P10	P041404	1–4	500	73	†	0.69
	P041408	1–3	500	83	†	0.86
	P041412	1–3	400	74	†	0.63
	P041413	1–3	400	79	†	0.69

Last updated 19 November 2009 (GF)