NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Arts (Visual Arts) Assessment, by Item and Block: 2008

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Arts (Visual Arts) Assessment, by Item and Block: 2008

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 arts (visual arts) assessment, by item and block: 2008
Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. Item statistics in this table are described on the page Blocks of Items. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2008 Arts (Visual Arts) Assessment.
V3	VC00002	1–3	500	88	—	0.90
	VC00005	1–4	500	84	—	0.81
	VC00007	1–4	400	69	—	0.81
	VC000A4	1–3	500	84	—	0.87
	VC000A6	1–2	500	98	0.90	—
	VC000B4	1–3	500	74	—	0.70
	VC000B6	1–4	500	75	—	0.70
V4	VD00001	1–2	500	95	0.90	—
	VD00007	1–3	500	77	—	0.67
	VD00008	1–4	500	76	—	0.69
	VD00009	1–4	500	79	—	0.84
	VD000A5	1–3	500	82	—	0.79
	VD000A6	1–3	500	76	—	0.72
	VD000B5	1–3	500	84	—	0.86
	VD000B6	1–2	500	87	0.75	—
V5	VE00001	1–2	500	86	0.71	—
	VE00002	1–4	500	82	—	0.77
	VE00303	1–3	500	76	—	0.67
	VE00004	1–3	500	81	—	0.76
	VE00008	1–3	400	80	—	0.83
	VE00009	1–5	500	84	—	0.88
	VE00010	1–3	400	86	—	0.82
V6	VF00001	1–3	500	83	—	0.79
	VF00004	1–3	500	84	—	0.83
	VF000A6	1–4	500	83	—	0.83
	VF000A7	1–3	400	92	—	0.93
	VF000B6	1–4	500	80	—	0.82
	VF000B7	1–2	500	91	0.79	—
	VF000B8	1–4	500	81	—	0.86
	VF000C6	1–4	500	79	—	0.79
	VF000C7	1–3	500	88	—	0.89

Last updated 01 November 2011 (JL)