NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 U.S. History Assessment: 2001

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 U.S. History Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 U.S. history assessment: 2001
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
H3	H044301	1–4	600	78	†	0.89
	H044501	1–4	600	90	†	0.96
	H044702	1–3	600	94	†	0.94
	H045102	1–3	600	92	†	0.96
	H045301	1–3	600	94	†	0.96
H4	H045501	1–4	600	81	†	0.87
	H045901	1–4	600	80	†	0.89
	H046001	1–2	600	99	0.99	†
	H046101	1–3	600	96	†	0.96
	H046301	1–3	600	95	†	0.96
H5	H062001	1–3	600	84	†	0.86
	H062201	1–3	600	92	†	0.94
	H063101	1–3	600	87	†	0.91
	H063401	1–4	600	87	†	0.95
	H063601	1–3	600	89	†	0.94
H6	H048901	1–3	600	99	†	1.00
	H049401	1–4	600	86	†	0.77
	H049503	1–3	600	86	†	0.88
	H049601	1–4	600	87	†	0.93
	H049701	1–3	600	98	†	0.99
H7	H050101	1–4	600	82	†	0.91
	H050201	1–4	600	82	†	0.91
	H051002	1–3	600	99	†	0.99
	H051101	1–3	600	87	†	0.89
	H051102	1–3	600	82	†	0.90
H8	H051301	1–3	600	86	†	0.89
	H052301	1–4	600	73	†	0.84
	H052501	1–3	600	92	†	0.96
	H052601	1–3	600	86	†	0.90
	H052701	1–3	600	88	†	0.91
H9	H060701	1–3	600	94	†	0.97
	H061501	1–3	600	82	†	0.86
	H061601	1–3	600	91	†	0.91
	H061801	1–3	600	96	†	0.96
H10	H042201	1–3	600	97	†	0.94
	H042801	1–4	600	92	†	0.95
	H042902	1–3	600	95	†	0.97
	H043001	1–3	600	86	†	0.90
	H043101	1–3	600	97	†	0.98
H11	H063801	1–3	600	82	†	0.80
	H064001	1–3	600	88	†	0.88
	H064101	1–3	600	89	†	0.93
	H064401	1–4	600	82	†	0.91
	H064901	1–3	600	84	†	0.86
	H065101	1–3	600	79	†	0.80
	H065201	1–3	600	78	†	0.85
	H065301	1–3	600	86	†	0.92
	H065401	1–4	600	88	†	0.95
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version