NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Civics Assessment, by Item and Block: 2006

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Civics Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 civics assessment, by item and block: 2006
Block	Item	Range of response codes	Sample size	Percent of exact agreement	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 Civics Assessment.
P3	P042171	1–3	600	91	†	0.90
	P042174	1–3	500	80	†	0.49
	P042177	1–3	600	88	†	0.89
	P042181	1–3	500	88	†	0.87
	P042184	1–3	500	86	†	0.80
P4	P040803	1–3	500	85	†	0.87
	P040807	1–4	600	91	†	0.94
	P040813	1–4	500	82	†	0.80
P5	P040903	1–3	500	68	†	0.64
	P040906	1–3	400	88	†	0.87
	P040910	1–3	500	85	†	0.77
	P040913	1–3	500	94	†	0.94
P6	P041003	1–3	500	89	†	0.90
	P041007	1–3	500	79	†	0.77
	P041013	1–3	600	83	†	0.74
	P041014	1–3	600	87	†	0.83
P7	P041102	1–3	500	88	†	0.88
	P041106	1–4	500	78	†	0.77
	P041111	1–3	500	81	†	0.80
	P041116	1–3	400	79	†	0.68
P8	P041202	1–3	600	80	†	0.80
	P041205	1–3	500	82	†	0.73
	P041213	1–4	500	86	†	0.89
P9	P041307	1–4	500	92	†	0.94
	P041309	1–3	500	87	†	0.87
	P041315	1–3	500	80	†	0.80
P10	P042152	1–3	600	92	†	0.93
	P042155	1–4	500	87	†	0.91
	P042158	1–3	600	84	†	0.80
	P042162	1–3	500	91	†	0.90

Last updated 19 November 2009 (GF)