NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 4 Science Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 4 Science Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 4 science assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
S3	K031001	1–3	300	95	†	0.89
	K031002	1–3	300	83	†	0.78
	K031003	1–3	300	84	†	0.73
	K031004	1–2	300	93	0.81	†
	K031005	1–3	300	82	†	0.76
	K031006	1–3	300	91	†	0.92
	K031007	1–3	300	84	†	0.86
S4	K031101	1–2	400	97	0.90	†
	K031102	1–2	400	94	0.87	†
	K031103	1–2	400	90	0.79	†
	K031104	1–2	400	98	0.84	†
	K031105	1–3	400	99	†	0.99
	K031107	1–4	400	90	†	0.94
S6	K031301	1–4	200	91	†	0.72
	K031302	1–3	200	96	†	0.80
	K031303	1–3	200	92	†	0.92
	K031304	1–3	200	92	†	0.92
	K031309	1–4	200	94	†	0.94
S7	K031401	1–4	600	78	†	0.54
	K031402	1–3	600	86	†	0.59
	K031403	1–3	600	84	†	0.57
	K031404	1–3	400	83	†	0.53
	K031407	1–3	600	75	†	0.46
	K031408	1–3	600	87	†	0.56
	K031409	1–3	600	85	†	0.63
	K031410	1–3	600	86	†	0.62
S9	K031602	1–3	600	89	†	0.81
	K031603	1–3	600	92	†	0.64
	K031604	1–3	600	92	†	0.82
	K031606	1–3	600	80	†	0.48
	K031607	1–4	600	78	†	0.70
	K031608	1–3	600	83	†	0.55
	K031609	1–3	600	87	†	0.65
S10	K031901	1–3	600	77	†	0.63
	K032001	1–3	600	87	†	0.74
	K032501	1–3	600	83	†	0.60
	K032502	1–3	600	87	†	0.53
	K032601	1–3	600	84	†	0.68
	K032602	1–3	600	84	†	0.64
S12	K034001	1–3	600	85	†	0.62
	K034101	1–3	600	83	†	0.72
	KW34101	1–3	600	89	†	0.91
	KX34101	1–3	600	88	†	0.67
	KY34101	1–3	600	75	†	0.67
	KZ34101	1–3	600	82	†	0.53
	KZ34401	1–3	600	82	†	0.67
	KZ34501	1–4	600	81	†	0.64
	KZ34502	1–3	600	89	†	0.81
S13	K034802	1–3	600	94	†	0.90
	K034901	1–3	600	86	†	0.54
	K034902	1–3	600	87	†	0.74
	K035201	1–3	600	80	†	0.73
	K035301	1–4	600	85	†	0.53
S14	K035601	1–3	600	81	†	0.56
	K035801	1–3	600	85	†	0.83
	K035901	1–3	600	83	†	0.60
	K036101	1–3	600	82	†	0.64
	K036301	1–3	600	86	†	0.76
S15	K037301	1–3	600	87	†	0.67
	K037401	1–3	600	81	†	0.24
	K037501	1–3	600	91	†	0.26
	K037601	1–3	600	79	†	0.58
	K037701	1–4	600	91	†	0.56
	K037702	1–3	600	83	†	0.58
S21	K039801	1–3	600	87	†	0.54
	K039901	1–5	400	78	†	0.69
	K040001	1–3	600	83	†	0.69
	K040301	1–3	600	80	†	0.64
	K040401	1–4	600	80	†	0.76
	K040501	1–3	600	84	†	0.64
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version