NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Mathematics Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Mathematics Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 mathematics assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
M3	M056801	1–2	800	99	0.99	†
	M056901	1–3	800	97	†	0.98
	M057001	1–2	800	100	0.99	†
M5	M070801	1–3	800	91	†	0.96
	M071001	1–3	800	98	†	0.98
	M071101	1–3	800	99	†	1.00
	M071201	1–3	800	98	†	0.99
	M071301	1–3	800	99	†	0.99
	M071401	1–5	800	98	†	0.99
M6	M020201	1–2	800	99	0.97	†
	M020301	1–4	800	99	†	0.98
	M020401	1–2	800	99	0.98	†
	M020501	1–2	800	100	0.99	†
	M020801	1–6	800	97	†	0.99
	M020901	1–2	800	95	0.92	†
	M021001	1–2	800	99	0.98	†
	M021101	1–3	800	95	†	0.97
	M021201	1–3	800	98	†	0.98
	M021401	1–2	800	100	1.00	†
	M021501	1–2	800	99	0.98	†
	M021502	1–2	800	100	0.99	†
	M021601	1–4	800	97	†	0.93
	M021602	1–2	800	98	0.96	†
	M021701	1–2	800	100	1.00	†
	M021702	1–2	800	99	0.98	†
	M021801	1–2	800	99	0.98	†
M7	M071502	1–3	800	96	†	0.98
	M071602	1–3	800	94	†	0.95
	M071603	1–3	800	92	†	0.95
	M071604	1–3	800	96	†	0.98
	M071701	1–3	800	98	†	0.98
	M071801	1–5	800	96	†	0.99
M8	M011931	1–2	800	99	0.98	†
	M012031	1–3	800	100	†	1.00
	M013031	1–4	800	100	†	1.00
	M013131	1–2	800	98	0.97	†
M9	M052401	1–2	800	94	0.90	†
M9	M053401	1–5	800	75	†	0.92
M10	M094201	1–4	800	97	†	0.99
	M094301	1–3	800	94	†	0.96
	M094701	1–5	800	88	†	0.96
M11	M058901	1–3	800	99	†	0.99
	M059702	1–3	800	96	†	0.98
	M059801	1–2	800	100	1.00	†
M12	M092301	1–3	800	97	†	0.98
	M092401	1–3	800	96	†	0.96
	M092601	1–3	800	98	†	0.99
	M092901	1–5	800	95	†	0.98
M13	M095001	1–3	800	95	†	0.98
	M095301	1–3	800	96	†	0.96
	M095401	1–4	800	99	†	1.00
M14	M073401	1–3	800	95	†	0.96
	M073801	1–3	800	99	†	0.99
	M073901	1–3	800	99	†	0.99
	M074001	1–3	800	95	†	0.98
	M074101	1–5	800	97	†	0.98
M15	M076101	1–3	800	99	†	0.98
	M076601	1–3	800	99	†	1.00
	M076701	1–3	800	99	†	0.99
	M076801	1–3	800	99	†	0.99
	M076901	1–3	800	99	†	0.98
	M077001	1–5	800	94	†	0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version