NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 8 Science Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 8 Science Assessment: 2000

K036401

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 8 science assessment: 2000
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
S3	K040601	1–3	600	98	†	0.97
	K040603	1–3	600	94	†	0.95
	K040604	1–3	600	85	†	0.80
	K040605	1–3	600	89	†	0.87
	K040606	1–4	600	93	†	0.94
	K040607	1–4	600	91	†	0.95
	K040608	1–4	600	84	†	0.93
	K040609	1–4	600	94	†	0.95
	K040610	1–4	600	90	†	0.90
S5	K040801	1–3	800	97	†	0.94
	K040802	1–3	800	98	†	0.94
	K040803	1–3	700	97	†	0.96
	K040804	1–3	500	89	†	0.68
	K040805	1–3	500	93	†	0.87
	K040806	1–2	400	87	0.58	†
	K040808	1–3	700	95	†	0.96
	K040809	1–3	600	92	†	0.91
S6	K031301	1–4	400	98	†	0.83
	K031302	1–3	400	89	†	0.79
	K031305	1–3	400	92	†	0.76
	K031306	1–3	400	97	†	0.74
	K031307	1–3	400	92	†	0.87
	K031308	1–3	600	98	†	0.94
	K031309	1–4	400	98	†	0.92
S8	K041306	1–3	600	81	†	0.85
	K041307	1–3	600	85	†	0.79
	K041401	1–3	600	93	†	0.92
	K041402	1–3	600	95	†	0.91
	K041403	1–3	600	89	†	0.77
S9	K031602	1–3	600	97	†	0.97
	K031603	1–3	600	97	†	0.96
	K031604	1–3	600	98	†	0.97
	K031606	1–3	600	88	†	0.84
	K031607	1–4	600	87	†	0.90
	K031608	1–3	600	94	†	0.91
	K031609	1–3	600	96	†	0.97
	K031610	1–3	600	96	†	0.95
	K031611	1–3	600	94	†	0.85
	K031613	1–3	600	98	†	0.91
S11	K043001	1–3	600	92	†	0.87
	K043101	1–3	600	87	†	0.88
	K043102	1–4	600	76	†	0.83
	K043103	1–3	600	90	†	0.73
	K043501	1–3	600	93	†	0.92
	K043601	1–3	600	91	†	0.88
	K043602	1–3	600	96	†	0.86
	K043603	1–3	600	92	†	0.80
S12	K047201	1–4	900	85	†	0.92
	K047301	1–3	800	95	†	0.95
	K047401	1–3	900	92	†	0.92
	K047901	1–3	800	91	†	0.90
	K048001	1–3	800	91	†	0.76
	K048101	1–4	700	84	†	0.76
	K048102	1–3	600	92	†	0.51
	K048103	1–2	700	87	0.72	†
S13	K048601	1–3	600	96	†	0.96
	K048901	1–3	600	97	†	0.98
	K049001	1–3	500	99	†	0.77
	K049301	1–3	500	97	†	0.83
	K049401	1–3	600	91	†	0.86
	K049402	1–3	500	84	†	0.86
	K049403	1–3	500	85	†	0.85
	K049404	1–4	500	76	†	0.74
S14	K035601	1–3	600	78	†	0.69
	K035801	1–3	600	93	†	0.94
	K035901	1–3	600	89	†	0.88
	K036101	1–3	600	89	†	0.83
	K036301	1–3	600	98	†	0.98
	K036401	1–3	600	79	†	0.18
	K036402	1–3	600	91	†	0.88
	K036403	1–3	600	66	†	0.09
	K036404	1–3	600	80	†	0.22
	K036701	1–3	600	96	†	0.89
	K036801	1–3	600	96	†	0.94
S15	K037301	1–3	600	94	†	0.95
	K037401	1–3	600	90	†	0.77
	K037501	1–3	600	97	†	0.85
	K037601	1–3	600	82	†	0.81
	K037701	1–4	600	99	†	0.99
	K037703	1–3	600	94	†	0.92
	K038101	1–2	600	97	0.83	†
	K038201	1–3	600	95	†	0.87
	K038301	1–5	400	87	†	0.90
S21	K045301	1–3	600	92	†	0.92
	K045601	1–4	600	97	†	0.97
	K045701	1–3	600	87	†	0.88
	K045801	1–3	600	92	†	0.87
	K046301	1–3	600	91	†	0.84
	K046401	1–3	600	94	†	0.82
	K046501	1–3	600	95	†	0.94
	K046601	1–3	600	85	†	0.87
	K046701	1–4	600	85	†	0.84
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Last updated 11 July 2008 (KL)

Printer-friendly Version