NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 12 U.S. History Assessment:

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 12 U.S. History Assessment: 2001

Range of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 12 U.S. history assessment: 2001
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
H3	H044301	1–4	600	70	†	0.75
	H044301	1–4	600	84	†	0.91
	H044702	1–3	600	91	†	0.81
	H045102	1–3	600	86	†	0.83
	H045301	1–3	600	89	†	0.88
H4	H045501	1–4	600	61	†	0.67
	H045901	1–4	600	83	†	0.85
	H046001	1–2	700	100	1.00	†
	H046101	1–3	600	96	†	0.95
	H046301	1–3	600	88	†	0.82
H6	H048901	1–3	600	97	†	0.98
	H049401	1–4	600	84	†	0.67
	H049503	1–3	600	85	†	0.77
	H049601	1–4	600	84	†	0.84
	H049701	1–3	600	99	†	0.99
H7	H050101	1–4	600	79	†	0.86
	H050201	1–4	600	76	†	0.78
	H051101	1–3	600	84	†	0.83
	H051102	1–3	600	78	†	0.82
H8	H051301	1–3	600	79	†	0.68
	H052301	1–4	600	82	†	0.81
	H052501	1–3	600	87	†	0.90
	H052601	1–3	600	81	†	0.73
H10	H042201	1–3	600	97	†	0.88
	H042801	1–4	600	89	†	0.94
	H042902	1–3	600	91	†	0.94
	H043001	1–3	600	81	†	0.84
	H043101	1–3	600	96	†	0.96
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version

Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.