NAEP Analysis and Scaling - Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 8 Mathematics Combined Nation

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 8 Mathematics Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2005, by block and item, grade 8 mathematics combined national and state main assessment: 2005
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
M3	M075401	1–3	1,200	93	†	0.91
	M075601	1–3	2,000	94	†	0.96
	M019901	1–3	400	99	†	0.97
	M047301	1–4	400	99	†	0.97
	M020101	1–2	800	99	0.98	†
	M076001	1–5	2,000	82	†	0.88
M4	M046001	1–5	400	99	†	0.97
	M046901	1–5	600	99	†	0.98
	M046601	1–4	1,000	96	†	0.92
	M046801	1–5	800	99	†	0.99
	M066601	1–3	800	92	†	0.94
	M067201	1–3	1,000	73	†	0.76
	M068003	1–3	400	99	†	1.00
	M068005	1–3	600	93	†	0.88
	M068008	1–3	1,400	89	†	0.84
	M068006	1–3	2,400	91	†	0.92
	M093601	1–3	1,000	97	†	0.96
	M053001	1–2	600	89	0.72	†
	M086301	1–5	1,400	92	†	0.96
M5	M085701	1–3	600	96	†	0.96
	M085901	1–3	1,800	95	†	0.95
	M019701	1–2	400	99	0.98	†
	M020001	1–2	200	97	0.94	†
	M066501	1–3	1,000	94	†	0.96
	M066301	1–3	1,400	96	†	0.89
	M067901	1–3	600	97	†	0.96
	M047901	1–3	400	99	†	0.98
	M053101	1–5	1,800	86	†	0.94
M6	M141301	1–4	1,100	923	†	0.97
	M141601	1–2	3,300	85	0.68	†
	M141901	1–5	2,400	90	†	0.96
M7	M144201	1–2	400	99	0.99	†
	M144601	1–3	1,100	93	†	0.94
	M144901	1–4	2,500	90	†	0.96
	M145101	1–4	1,300	96	†	0.94
M8	M021001	1–2	400	100	1.00	†
	M020901	1–2	600	89	0.77	†
	M092401	1–3	800	95	†	0.91
	M092601	1–3	800	96	†	0.94
	M073601	1–5	2,200	81	†	0.94
M9	M013131	1–2	400	98	0.96	†
	M073501	1–3	900	97	†	0.95
	M052401	1–2	1,500	89	0.77	†
	M075301	1–3	400	96	†	0.94
	M072901	1–3	1,000	93	†	0.94
	M075801	1–3	1,700	82	†	0.86
	M093801	1–5	1,000	91	†	0.96
M10	M142801	1–2	500	99	0.98	†
	M143201	1–2	500	99	0.98	†
	M143501	1–5	3,200	82	†	0.94
M11	M106301	1–2	400	99	0.99	†
	M106901	1–2	2,600	84	0.66	†
	M107501	1–2	400	99	0.99	†
M12	M110201	1–2	600	98	0.93	†
	M110501	1–3	1,400	90	†	0.91
	M111801	1–3	2,800	87	†	0.90
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Mathematics Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version