NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 8 Mathematics Assessment: 2

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 8 Mathematics Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 8 mathematics assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
M5	M066301	1–3	600	98	†	0.95
	M066501	1–3	600	93	†	0.93
	M066601	1–3	600	96	†	0.97
	M067201	1–3	600	86	†	0.83
	M067501	1–5	300	79	†	0.63
M6	M019701	1–2	600	98	0.92	†
	M019801	1–3	700	97	†	0.94
	M019901	1–3	700	98	†	0.94
	M020001	1–2	600	99	0.98	†
	M020101	1–2	600	100	1.00	†
	M020201	1–2	600	88	0.54	†
	M020301	1–4	600	99	†	0.99
	M020401	1–2	600	100	1.00	†
	M020501	1–2	600	99	0.98	†
	M020801	1–6	600	96	†	0.99
	M020901	1–2	600	40	0.26	†
	M021001	1–2	600	100	0.99	†
	M021101	1–3	700	91	†	0.92
	M021201	1–3	600	98	†	0.97
	M021301	1–2	600	97	0.94	†
	M021302	1–2	600	95	0.89	†
M7	M067901	1–3	600	99	†	0.99
	M068003	1–3	600	100	†	0.99
	M068005	1–3	600	98	†	0.96
	M068006	1–3	300	91	†	0.95
	M068008	1–3	600	93	†	0.89
	M068201	1–5	600	88	†	0.89
M8	M013031	1–4	600	100	†	1.00
M8	M013131	1–2	600	98	0.96	†
M9	M052401	1–2	600	92	0.82	†
	M053001	1–2	600	93	0.82	†
	M053101	1–4	600	30	†	0.00
	M053101	1–4	400	81	†	0.85
M11	M046001	1–5	600	100	†	0.99
	M046601	1–4	600	93	†	0.91
	M046801	1–5	600	100	†	1.00
	M046901	1–5	600	99	†	1.00
	M047301	1–4	600	99	†	0.95
	M047901	1–3	600	100	†	1.00
M13	M051201	1–2	600	99	0.97	†
	M051301	1–2	600	100	0.98	†
	M051601	1–2	600	99	0.98	†
	M052101	1–3	600	96	†	0.88
	M052201	1–4	500	87	†	0.92
M14	M072901	1–3	600	92	†	0.94
	M073401	1–3	600	97	†	0.95
	M073501	1–3	600	98	†	0.97
	M073601	1–5	400	85	†	0.94
M15	M075301	1–3	600	97	†	0.95
	M075401	1–3	600	92	†	0.91
	M075601	1–3	600	90	†	0.93
	M075801	1–3	600	79	†	0.81
	M076001	1–5	500	85	†	0.88
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version