NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2003, by Block and Item, Grade 4 Reading Combi

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2003, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2003

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous assessment years, rescored in 2003, by block and item, grade 4 reading combined national and state main assessment: 2003
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017001	1–2	1,400	88	0.74	†
	R017003	1–3	2,000	86	†	0.86
	R017004	1–2	1,400	94	0.85	†
	R017006	1–2	1,600	92	0.78	†
	R017007	1–4	2,200	73	†	0.86
	R017009	1–3	3,000	85	†	0.78
R4	R012102	1–2	600	94	0.88	†
	R012104	1–2	1000	93	0.86	†
	R012106	1–2	1,000	86	0.72	†
	R012109	1–2	200	89	0.74	†
	R012111	1–4	3,000	89	†	0.92
	R012112	1–2	1,000	87	0.73	†
	R012113	1–3	800	88	†	0.87
R5	R012601	1–2	1000	85	0.68	†
	R012604	1–2	2,000	93	0.84	†
	R012607	1–4	3,600	80	†	0.83
	R012612	1–2	3,000	83	0.67	†
R6	R017301	1–2	2,400	90	0.80	†
	R017303	1–3	2,000	86	†	0.88
	R017307	1–4	4,200	77	†	0.86
	R017309	1–3	3,200	83	†	0.84
	R017310	1–3	2,000	85	†	0.84
R7	R012702	1–2	1,000	91	0.79	†
	R012703	1–2	800	87	0.74	†
	R012705	1–2	2,000	91	0.78	†
	R012706	1–2	1,200	80	0.56	†
	R012710	1–2	600	89	0.76	†
	R012714	1–4	3,800	80	†	0.82
R8	R017401	1–3	2,000	83	†	0.85
	R017701	1–3	1,200	85	†	0.87
	R017901	1–4	4,400	79	†	0.89
	R018201	1–3	3,400	85	†	0.87
	R018301	1–3	2,800	84	†	0.85
R9	R015803	1–3	1,800	85	†	0.83
	R015804	1–4	3,200	78	†	0.81
	R015806	1–3	2,600	81	†	0.81
	R015807	1–3	1,400	82	†	0.81
	R015809	1–3	1,800	87	†	0.87
	R015810	1–3	2,200	88	†	0.87
R10	R012503	1–2	600	87	0.75	†
	R012504	1–2	600	97	0.93	†
	R012506	1–2	600	89	0.78	†
	R012508	1–2	600	97	0.94	†
	R012511	1–2	600	93	0.85	†
	R012512	1–4	2,800	76	†	0.86
R11	R020401	1–3	3,200	86	†	0.88
	R020701	1–2	800	88	0.77	†
	R021201	1–3	2,400	84	†	0.86
	R021701	1–3	2,200	79	†	0.81
R12	R023201	1–2	1,200	91	0.82	†
	R023501	1–3	2,200	85	†	0.89
	R023601	1–3	1,200	84	†	0.87
	R024001	1–3	3,600	80	†	0.82
†The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Reading Assessment.

Last updated 25 March 2009 (GF)

Printer-friendly Version