NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Asses

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Assessment: 2003

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 reading combined national and state main assessment: 2003
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	1,200	96	0.92	†
	R017102	1–3	1,200	93	†	0.94
	R017104	1–3	1,100	97	†	0.97
	R017105	1–4	1,100	87	†	0.94
	R017107	1–3	1,100	93	†	0.96
	R017108	1–2	1,100	96	0.91	†
	R017110	1–3	1000	96	†	0.97
R4	R018401	1–3	1,200	87	†	0.90
	R018601	1–3	1,100	78	†	0.77
	R018701	1–3	1,200	85	†	0.90
	R018801	1–3	1,200	90	†	0.92
	R019001	1–4	1,200	82	†	0.92
	R019101	1–3	1,100	91	†	0.91
	R019201	1–3	1000	75	†	0.76
R5	R012601	1–2	1,200	94	0.88	†
	R012604	1–2	1,200	93	0.87	†
	R012607	1–4	1,200	82	†	0.89
	R012612	1–2	1,100	84	0.69	†
R6	R013201	1–4	1,200	83	†	0.90
	R013203	1–2	1,200	99	0.92	†
	R013205	1–2	1,200	97	0.90	†
	R013207	1–2	1,200	94	0.87	†
	R013209	1–2	1,100	96	0.92	†
	R013211	1–2	1,000	92	0.85	†
	R013212	1–4	1,100	88	†	0.92
R7	R012702	1–2	1,200	96	0.80	†
	R012703	1–2	1,200	87	0.72	†
	R012705	1–2	1,200	88	0.78	†
	R012706	1–2	1,200	86	0.73	†
	R012710	1–2	1,100	89	0.75	†
	R012713	1–2	1000	98	0.96	†
	R012714	1–4	1,200	80	†	0.87
R8	R017202	1–3	1,200	92	†	0.93
	R017204	1–3	1,200	89	†	0.90
	R017205	1–4	1,200	79	†	0.89
	R017207	1–3	1,200	83	†	0.86
	R017208	1–3	1,100	89	†	0.89
	R017210	1–2	1,000	92	0.81	†
R9	R016101	1–3	1,100	86	†	0.87
	R016104	1–3	1,100	83	†	0.79
	R016107	1–3	1,100	95	†	0.97
	R016108	1–3	1,100	74	†	0.79
	R016109	1–3	1,100	87	†	0.83
R10	R013402	1–2	1,200	98	0.96	†
	R013403	1–4	1,200	97	†	0.97
	R013405	1–2	1,200	93	0.87	†
	R013406	1–4	1,200	82	†	0.93
	R013407	1–2	1,200	94	0.89	†
	R013409	1–2	1,100	94	0.89	†
	R013411	1–2	1,000	94	0.86	†
	R013413	1–2	1,000	88	0.76	†
R11	R013001	1–2	1,200	95	0.83	†
	R013003	1–2	1,200	100	0.99	†
	R013004	1–4	1,200	83	†	0.93
	R013005	1–2	1,200	92	0.76	†
	R013007	1–2	1,100	97	0.92	†
	R013008	1–2	1,100	87	0.77	†
	R013009	1–2	1,100	93	0.78	†
	R013010	1–2	1,100	96	0.88	†
	R013011	1–2	1,000	86	0.73	†
R12	R024401	1–3	1,100	92	†	0.89
	R024501	1–3	1,100	89	†	0.85
	R024801	1–4	1,200	79	†	0.84
	R025001	1–3	1,200	93	†	0.94
	R025601	1–3	1,100	81	†	0.73
	R026101	1–2	1,100	86	0.69	†
R13	R016201	1–3	1,200	95	†	0.89
	R016202	1–3	1,200	94	†	0.92
	R016204	1–4	1,200	91	†	0.87
	R016205	1–3	1,200	93	†	0.93
	R016207	1–3	1,200	97	†	0.98
	R016210	1–4	1,200	85	†	0.92
	R016211	1–3	1,100	91	†	0.86
	R016212	1–3	1,100	92	†	0.94
	R016213	1–3	1,100	92	†	0.86
R14	R026501	1–3	1,200	87	†	0.90
	R026801	1–4	1,200	84	†	0.92
	R027001	1–2	1,100	91	0.74	†
	R027201	1–3	1,200	92	†	0.94
R15	R028401	1–3	1,200	93	†	0.92
	R028501	1–3	1,200	94	†	0.93
	R028801	1–4	1,200	84	†	0.90
	R029601	1–3	1,100	89	†	0.89
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Reading Assessment.

Last updated 25 March 2009 (GF)

Printer-friendly Version