NAEP Analysis and Scaling - Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Mathematics Combined National and State Main Asse

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Mathematics Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 mathematics combined national and state main assessment: 2005
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
M3	M019901	1–3	1,600	98	†	0.97
	M020101	1–2	1,500	99	0.98	†
	M043501	1–5	1,400	98	†	0.99
	M047301	1–4	1,600	98	†	0.95
	M074301	1–2	1,600	99	0.98	†
	M074501	1–3	1,600	92	†	0.94
	M074801	1–3	1,600	98	†	0.98
	M074901	1–3	1,600	96	†	0.97
	M075001	1–3	1,600	99	†	0.98
	N277903	1–2	1,600	100	0.98	†
M4	M046001	1–5	1,700	100	†	1.00
	M046601	1–4	1,700	95	†	0.95
	M046801	1–5	1,700	99	†	0.98
	M046901	1–5	1,700	99	†	1.00
	M066601	1–3	1,700	93	†	0.94
	M068001	1–3	1,700	99	†	0.99
	M068002	1–3	1,600	98	†	0.98
	M068003	1–3	1,400	99	†	0.99
	M068004	1–5	1,300	88	†	0.95
M5	M019701	1–2	1,500	99	0.98	†
	M020001	1–2	1,500	99	0.98	†
	M066301	1–3	1,200	96	†	0.94
	M066501	1–3	1,300	98	†	0.99
	M067901	1–3	1,100	97	†	0.98
	M085401	1–5	1,600	91	†	0.94
	M085701	1–3	1,500	96	†	0.96
	M085901	1–3	1,600	94	†	0.94
M6	M136501	1–2	1,800	99	0.98	†
	M136601	1–3	1,800	94	†	0.96
	M136901	1–4	1,800	97	†	0.99
M7	M139301	1–2	1,800	99	0.97	†
	M139601	1–3	1,700	98	†	0.98
	M139801	1–4	1,700	92	†	0.96
	M140001	1–3	1,600	96	†	0.95
	M140301	1–5	1,300	94	†	0.97
M8	M010631	1–3	1,700	98	†	0.98
	M040001	1–3	1,600	98	†	0.98
	M072401	1–3	1,200	96	†	0.97
	M087001	1–3	1,700	98	†	0.99
	M087301	1–5	1000	92	†	0.94
	M091201	1–3	1,400	95	†	0.97
M9	M040201	1–2	1,600	95	0.83	†
	M043201	1–2	1,700	99	0.97	†
	M043301	1–3	1,600	99	†	0.99
	M043401	1–4	1,700	96	†	0.98
	M043402	1–4	1,700	98	†	0.99
	M043403	1–3	1,600	99	†	0.98
	M066701	1–3	1,700	97	†	0.97
	M066801	1–3	1,700	97	†	0.96
	M086601	1–3	1,700	98	†	0.96
	M091401	1–5	1,500	91	†	0.96
M10	M137501	1–2	1,900	100	0.99	†
	M138201	1–2	1,900	96	0.91	†
	M138401	1–4	1,800	87	†	0.95
	M138701	1–5	1,700	92	†	0.98
M11	M096001	1–2	1,600	97	0.94	†
	M097201	1–3	1,600	97	†	0.98
	M097401	1–3	1,600	99	†	0.98
M12	M100201	1–2	1,700	100	1.00	†
	M100501	1–2	1,700	99	0.99	†
	M101401	1–2	1,700	99	0.97	†
	M101501	1–3	1,700	98	†	0.98
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Mathematics Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version