NAEP Analysis and Scaling - Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Science National Main Assessment: 2005

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Science National Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 science national main assessment: 2005
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B3	K049501	1–4	500	98	†	0.82
	K049502	1–5	500	92	†	0.95
	K049503	1–3	500	85	†	0.88
	K049504	1–3	500	87	†	0.90
	K049505	1–3	500	88	†	0.88
	K049506	1–3	400	87	†	0.67
B5	K040801	1–3	500	97	†	0.89
	K040802	1–3	500	99	†	0.95
	K040803	1–3	500	97	†	0.96
	K040804	1–3	400	91	†	0.92
	K040805	1–3	400	94	†	0.94
	K040806	1–2	400	87	0.69	†
	K040808	1–3	500	97	†	0.98
	K040809	1–3	400	95	†	0.97
B6	K049701	1–3	500	99	†	0.99
	K049702	1–3	500	98	†	0.98
	K049703	1–3	500	93	†	0.93
	K049704	1–3	500	96	†	0.95
	K049705	1–4	400	88	†	0.91
	K049706	1–3	400	83	†	0.89
	K049707	1–5	400	82	†	0.93
	K049708	1–2	500	96	0.92	†
B7	K104501	1–3	600	91	†	0.84
	K104601	1–3	600	70	†	0.52
	K105001	1–3	600	90	†	0.88
	K105501	1–3	500	93	†	0.83
	K105601	1–2	600	96	0.89	†
	K106101	1–3	600	93	†	0.60
B8	K041306	1–3	700	83	†	0.85
	K041307	1–3	700	82	†	0.84
	K041401	1–3	700	90	†	0.84
	K041402	1–3	600	93	†	0.89
	K041403	1–3	600	86	†	0.79
	K041404	1–3	700	92	†	0.93
	K041406	1–3	600	92	†	0.81
B10	K090301	1–3	400	88	†	0.86
	K090501	1–3	400	98	†	0.96
	K092101	1–3	300	84	†	0.76
	K092701	1–3	400	92	†	0.91
	K094601	1–3	400	92	†	0.72
	K098601	1–3	500	88	†	0.78
B12	K047201	1–4	500	86	†	0.91
	K047301	1–3	400	84	†	0.85
	K047401	1–3	400	92	†	0.94
	K047901	1–3	400	89	†	0.90
	K048001	1–3	400	97	†	0.92
B13	K048601	1–3	400	87	†	0.85
	K048901	1–3	400	96	†	0.97
	K049001	1–3	400	97	†	0.89
	K049301	1–3	400	98	†	0.91
	K049401	1–3	400	89	†	0.86
	K049402	1–3	400	82	†	0.85
	K049403	1–3	400	80	†	0.70
	K049404	1–4	400	72	†	0.71
B14	K052901	1–4	400	96	†	0.96
	K053001	1–3	400	87	†	0.78
	K053601	1–5	400	97	†	0.97
	K053701	1–3	400	96	†	0.94
	K053801	1–3	400	89	†	0.91
	K053901	1–3	400	95	†	0.89
B20	K092001	1–3	400	87	†	0.88
	K096501	1–3	300	94	†	0.75
	K100001	1–2	400	99	0.97	†
	K100101	1–3	400	88	†	0.86
	K100201	1–3	400	79	†	0.73
	K100301	1–3	400	91	†	0.78
B21	K059001	1–3	500	89	†	0.87
	K059101	1–3	400	94	†	0.90
	K059201	1–4	400	90	†	0.92
	K059801	1–3	400	88	†	0.82
	K059901	1–4	400	89	†	0.90
	K060001	1–3	400	83	†	0.85
	K060101	1–4	400	77	†	0.68
S9	K097901	1–3	500	80	†	0.75
	K098501	1–3	600	87	†	0.89
	K101801	1–3	600	81	†	0.80
	K102001	1–4	600	73	†	0.77
	K102301	1–3	700	95	†	0.94
S15	K075401	1–4	500	93	†	0.92
	K075501	1–3	300	94	†	0.93
	K075601	1–3	400	87	†	0.85
	K075701	1–3	400	94	†	0.90
	K075801	1–3	400	93	†	0.84
	K075901	1–3	300	99	†	0.98
S16	K072201	1–4	400	79	†	0.85
S16	K072601	1–4	500	97	†	0.98
S17	K073501	1–4	400	80	†	0.83
S17	K073801	1–3	500	93	†	0.90
S18	K074301	1–4	500	90	†	0.85
	K074501	1–3	400	99	†	0.99
	K074601	1–3	400	92	†	0.93
	K074801	1–4	400	78	†	0.87
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version