NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 12 Geography Assessment: 20

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 12 Geography Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 12 geography assessment: 2001
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
G3	G020701	1–3	600	88	†	0.90
	G021001	1–4	600	79	†	0.81
	G021401	1–4	600	85	†	0.94
	G021601	1–4	600	82	†	0.79
	G021602	1–3	600	94	†	0.93
G5	G016201	1–3	600	98	†	0.98
	G016302	1–3	600	96	†	0.95
	G016401	1–3	600	94	†	0.95
	G016502	1–3	600	92	†	0.93
	G016701	1–3	600	91	†	0.92
	G017101	1–4	600	80	†	0.86
G8	G025001	1–3	600	88	†	0.85
	G025101	1–4	600	75	†	0.84
	G025202	1–3	600	83	†	0.79
	G025301	1–4	600	92	†	0.86
	G025601	1–3	600	93	†	0.84
	G025801	1–3	600	90	†	0.87
G9	G026101	1–3	600	94	†	0.92
	G026204	1–3	600	91	†	0.94
	G026301	1–4	600	90	†	0.84
	G026502	1–3	600	97	†	0.97
	G026601	1–3	600	96	†	0.95
	G026901	1–3	600	92	†	0.93
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.

Last updated 11 July 2008 (KL)