NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 U.S. History Assessment: 2006

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 U.S. History Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 4 U.S. history assessment, by item and block: 2006
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
H3	H065701	1–4	500	77.47	†	0.72
	H065801	1–4	500	97.98	†	0.99
	H066001	1–3	400	77.23	†	0.75
	H066501	1–4	400	97.05	†	0.98
	H066701	1–3	400	93.98	†	0.90
H4	H054301	1–3	500	86.42	†	0.87
	H054601	1–3	500	91.19	†	0.88
	H054801	1–3	500	90.82	†	0.91
	H054901	1–3	500	94.94	†	0.95
	H055301	1–3	400	91.33	†	0.91
H5	H055901	1–3	600	90.23	†	0.91
	H056101	1–3	400	93.43	†	0.93
	H056401	1–3	500	95.84	†	0.96
	H056601	1–3	500	95.26	†	0.95
	H056801	1–3	500	97.40	†	0.95
H6	H067401	1–3	500	91.72	†	0.91
	H067701	1–3	500	95.73	†	0.96
	H068001	1–4	500	83.84	†	0.83
	H068201	1–3	400	94.64	†	0.93
	H068301	1–3	400	92.00	†	0.90
	H068501	1–3	300	87.19	†	0.83
H7	H057501	1–3	500	97.35	†	0.96
	H057701	1–3	500	87.87	†	0.83
	H057801	1–3	400	88.50	†	0.84
	H058601	1–3	400	84.44	†	0.85
	H058701	1–3	400	82.92	†	0.80
H8	H034101	1–4	500	96.13	†	0.82
	H034401	1–3	500	96.22	†	0.95
	H034501	1–2	500	99.36	0.96	†
	H034702	1–3	400	98.41	†	0.93
	H035001	1–3	400	86.17	†	0.84
	H035101	1–3	500	88.24	†	0.90
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 U.S. History Assessment.

Last updated 19 November 2009 (GF)

Printer-friendly Version