NAEP Analysis and Scaling - Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Reading National Main Assessment: 2005

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Reading National Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 reading national main assessment: 2005
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	500	94	0.84	†
	R017102	1–3	400	83	†	0.85
	R017104	1–3	400	87	†	0.88
	R017105	1–4	400	71	†	0.80
	R017107	1–3	400	86	†	0.86
	R017108	1–2	400	97	0.93	†
	R017110	1–3	400	94	†	0.94
R4	R013501	1–2	400	92	0.84	†
	R013503	1–2	400	92	0.83	†
	R013505	1–2	400	90	0.78	†
	R013506	1–4	300	73	†	0.75
	R013508	1–2	400	90	0.80	†
	R013509	1–2	300	90	0.78	†
R5	R016301	1–3	400	86	†	0.81
	R016302	1–3	400	89	†	0.85
	R016303	1–3	400	87	†	0.86
	R016305	1–3	400	81	†	0.84
	R016306	1–3	400	84	†	0.87
	R016307	1–3	300	88	†	0.85
	R016308	1–4	300	86	†	0.88
R6	R013201	1–4	400	84	†	0.91
	R013203	1–2	400	97	0.60	†
	R013205	1–2	400	97	0.74	†
	R013207	1–2	400	91	0.74	†
	R013209	1–2	400	96	0.92	†
	R013211	1–2	400	86	0.72	†
	R013212	1–4	300	81	†	0.78
R7	R013701	1–2	500	89	0.78	†
	R013702	1–2	500	87	0.75	†
	R013704	1–2	400	91	0.68	†
	R013706	1–4	400	75	†	0.66
	R013708	1–2	400	91	0.81	†
	R013710	1–2	400	90	0.73	†
	R013712	1–2	400	83	0.65	†
R8	R053501	1–3	400	70	†	0.59
	R053504	1–2	400	99	0.95	†
	R053507	1–4	400	83	†	0.85
	R053509	1–2	400	93	0.75	†
R9	R053301	1–3	500	96	†	0.93
	R053303	1–2	400	94	0.85	†
	R053305	1–3	400	91	†	0.92
	R053309	1–3	400	87	†	0.89
R10	R013402	1–2	400	98	0.96	†
	R013403	1–4	400	93	†	0.96
	R013405	1–2	400	96	0.89	†
	R013406	1–4	400	78	†	0.91
	R013407	1–2	400	92	0.80	†
	R013409	1–2	400	95	0.84	†
	R013411	1–2	400	92	0.78	†
	R013413	1–2	400	86	0.70	†
R11	R019301	1–4	400	92	†	0.94
	R019401	1–3	400	90	†	0.91
	R019501	1–2	400	89	0.77	†
	R019701	1–3	400	92	†	0.94
	R020001	1–3	400	79	†	0.72
	R020201	1–3	400	87	†	0.86
R13	R015503	1–2	500	95	0.80	†
	R015505	1–2	400	88	0.76	†
	R015507	1–4	400	73	†	0.72
	R015509	1–2	400	89	0.77	†
	R015512	1–2	400	90	0.80	†
	R015514	1–4	400	71	†	0.70
R14	R016501	1–3	500	81	†	0.65
	R016502	1–3	500	88	†	0.81
	R016601	1–3	500	76	†	0.68
	R016602	1–3	500	86	†	0.86
	R016603	1–3	500	84	†	0.83
	R016604	1–3	500	84	†	0.80
	R016605	1–3	500	81	†	0.65
	R016701	1–4	400	85	†	0.76
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

Last updated 14 September 2009 (GF)

Printer-friendly Version