NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1998 That Were Rescored in 2000, by Block and Item, Grade 4 Reading Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1998 That Were Rescored in 2000, by Block and Item, Grade 4 Reading Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1998 that were rescored in 2000, by block and item, grade 4 reading assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa>	Intraclass correlation
R3	R017001	1–2	600	87	0.71	†
	R017003	1–3	600	71	†	0.74
	R017004	1–2	600	87	0.68	†
	R017006	1–2	600	92	0.78	†
	R017007	1–4	600	75	†	0.86
	R017009	1–3	600	83	†	0.73
R4	R012102	1–2	600	96	0.93	†
	R012104	1–2	600	94	0.87	†
	R012106	1–2	600	85	0.70	†
	R012108	1–2	600	94	0.85	†
	R012109	1–2	600	94	0.82	†
	R012111	1–4	600	86	†	0.88
	R012112	1–2	600	92	0.81	†
R5	R012601	1–2	600	89	0.69	†
	R012604	1–2	600	93	0.81	†
	R012607	1–4	600	81	†	0.83
	R012611	1–2	600	85	0.70	†
R7	R012702	1–2	600	92	0.79	†
	R012703	1–2	600	89	0.76	†
	R012705	1–2	600	91	0.68	†
	R012706	1–2	600	80	0.49	†
	R012708	1–4	600	80	†	0.77
	R012710	1–2	600	92	0.80	†
R8	R015702	1–3	600	88	†	0.81
	R015703	1–3	600	88	†	0.83
	R015704	1–3	600	79	†	0.81
	R015705	1–3	600	94	†	0.94
	R015707	1–4	600	92	†	0.91
	R015709	1–3	600	85	†	0.83
R9	R015802	1–2	600	91	0.79	†
	R015803	1–3	600	87	†	0.81
	R015804	1–4	600	74	†	0.73
	R015806	1–3	600	84	†	0.80
	R015807	1–3	600	76	†	0.73
	R015809	1–3	600	87	†	0.83
R10	R012503	1–2	600	88	0.77	†
	R012504	1–2	600	98	0.96	†
	R012506	1–2	600	90	0.79	†
	R012508	1–2	600	97	0.94	†
	R012511	1–2	600	94	0.85	†
	R012512	1–4	600	86	†	0.92
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Reading Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version