NAEP Analysis and Scaling - Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Assessme

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 reading combined national and state main assessment: 2005
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	1,200	93	0.86	†
	R017102	1–3	1,000	88	†	0.86
	R017104	1–3	1,100	95	†	0.94
	R017105	1–4	1,100	77	†	0.82
	R017107	1–3	900	90	†	0.89
	R017108	1–2	1,000	96	0.83	†
	R017110	1–3	1,000	95	†	0.95
R4	R018401	1–3	1,200	84	†	0.84
	R018601	1–3	1,200	77	†	0.76
	R018701	1–3	1,100	80	†	0.81
	R018801	1–3	1,100	84	†	0.85
	R019001	1–4	1,100	73	†	0.77
	R019101	1–3	1,100	83	†	0.81
	R019201	1–3	1,000	81	†	0.81
R5	R012601	1–2	1,200	91	0.82	†
	R012604	1–2	1,100	96	0.92	†
	R012607	1–4	1,100	78	†	0.83
	R012612	1–2	1,100	90	0.79	†
R6	R013201	1–4	1,200	89	†	0.93
	R013203	1–2	1,200	96	0.72	†
	R013205	1–2	1,100	96	0.80	†
	R013207	1–2	1,100	90	0.76	†
	R013209	1–2	1,100	96	0.88	†
	R013211	1–2	1,000	86	0.72	†
	R013212	1–4	900	82	†	0.77
R7	R012702	1–2	1,200	94	0.71	†
	R012703	1–2	1,200	85	0.70	†
	R012705	1–2	1,100	87	0.74	†
	R012706	1–2	1,200	85	0.68	†
	R012710	1–2	1,100	93	0.81	†
	R012713	1–2	1,000	98	0.96	†
	R012714	1–4	1,100	75	†	0.79
R8	R053401	1–3	1,200	85	†	0.87
	R053402	1–2	1,200	91	0.60	†
	R053405	1–4	1,100	76	†	0.77
	R053408	1–3	1,100	87	†	0.86
	R053410	1–2	1,100	91	0.77	†
R9	R053301	1–3	1,200	93	†	0.87
	R053303	1–2	1,100	90	0.80	†
	R053305	1–3	1,100	86	†	0.90
	R053309	1–3	1,100	88	†	0.90
R10	R013402	1–2	1,200	99	0.98	†
	R013403	1–4	1,100	96	†	0.96
	R013405	1–2	1,100	95	0.89	†
	R013406	1–4	1,100	81	†	0.92
	R013407	1–2	1,100	96	0.93	†
	R013409	1–2	1,000	93	0.85	†
	R013411	1–2	1,000	91	0.79	†
	R013413	1–2	1,000	84	0.67	†
R11	R013001	1–2	1,200	95	0.85	†
	R013003	1–2	1,200	100	0.99	†
	R013004	1–4	1,100	83	†	0.92
	R013005	1–2	1,100	91	0.75	†
	R013007	1–2	1,100	97	0.89	†
	R013008	1–2	1,100	87	0.75	†
	R013009	1–2	1,000	91	0.67	†
	R013010	1–2	1,000	97	0.90	†
	R013011	1–2	1,000	90	0.77	†
R12	R053202	1–3	1,200	88	†	0.86
	R053203	1–3	1,100	86	†	0.84
	R053205	1–4	1,100	77	†	0.87
	R053207	1–3	1,000	84	†	0.81
	R053209	1–3	1,100	89	†	0.87
R13	R016201	1–3	800	92	†	0.67
	R016202	1–3	800	90	†	0.80
	R016204	1–4	800	91	†	0.83
	R016205	1–3	800	82	†	0.70
	R016207	1–3	700	95	†	0.96
	R016210	1–4	700	76	†	0.76
	R016211	1–3	800	86	†	0.76
	R016212	1–3	800	88	†	0.90
	R016213	1–3	800	86	†	0.74
R14	R026501	1–3	1,200	84	†	0.83
	R026801	1–4	1,100	78	†	0.84
	R027001	1–2	1,000	89	0.69	†
	R027201	1–3	1,100	87	†	0.84
R15	R028401	1–3	1,200	94	†	0.92
	R028501	1–3	1,200	93	†	0.91
	R028801	1–4	1,100	85	†	0.89
	R029601	1–3	1,100	87	†	0.80
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version