NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2004, by Block and Item, Age 13 Mathematics Long-Term Tr

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2004, by Block and Item, Age 13 Mathematics Long-Term Trend Assessment: 2004

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2004, by block and item, age 13 mathematics long-term trend assessment: 2004
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B7	K104501	1–3	107	93	†	0.86
	K104601	1–3	100	68	†	0.41
	K105001	1–3	96	91	†	0.90
	K105501	1–3	73	100	†	1.00
	K105601	1–2	107	94	0.84	†
	K106101	1–3	98	97	†	0.48
B8	K041306	1–3	120	79	†	0.82
	K041307	1–3	116	84	†	0.85
	K041401	1–3	120	90	†	0.85
	K041402	1–3	113	89	†	0.85
	K041403	1–3	114	89	†	0.86
	K041404	1–3	113	89	†	0.88
	K041406	1–3	113	96	†	0.88
B9	K049901	1–3	555	92	†	0.92
	K049902	1–3	545	90	†	0.88
	K049903	1–3	500	93	†	0.89
	K049904	1–3	503	94	†	0.92
	K049907	1–3	534	93	†	0.93
	K049908	1–3	502	94	†	0.93
	K049909	1–4	414	98	†	0.96
	K049911	1–3	458	88	†	0.87
	K049912	1–5	417	77	†	0.89
	K049914	1–3	448	97	†	0.97
B10	K090301	1–3	76	92	†	0.92
	K090501	1–3	83	99	†	0.97
	K092101	1–3	51	98	†	0.98
	K092701	1–3	84	94	†	0.93
	K094601	1–3	77	94	†	0.52
	K098601	1–3	84	88	†	0.80
B11	K051701	1–2	412	90	0.80	†
	K051801	1–3	421	80	†	0.80
	K052301	1–3	362	88	†	0.82
	K052401	1–4	340	91	†	0.95
	K052402	1–3	333	96	†	0.91
	K052501	1–4	392	94	†	0.97
	K052502	1–4	412	96	†	0.97
	K052503	1–3	412	92	†	0.91
B12	K047201	1–4	87	85	†	0.91
	K047301	1–3	83	93	†	0.92
	K047401	1–3	83	89	†	0.88
	K047901	1–3	79	87	†	0.90
	K048001	1–3	76	95	†	0.88
	K048101	1–4	356	81	†	0.83
	K048102	1–3	328	93	†	0.64
	K048103	1–2	326	89	0.79	†
B13	K048601	1–3	88	86	†	0.87
	K048901	1–3	91	92	†	0.94
	K049001	1–3	84	99	†	0.96
	K049301	1–3	83	95	†	0.81
	K049401	1–3	83	93	†	0.87
	K049402	1–3	75	79	†	0.78
	K049403	1–3	83	80	†	0.67
	K049404	1–4	81	72	†	0.69
B14	K052901	1–4	117	95	†	0.98
	K053001	1–3	97	91	†	0.84
	K053101	1–3	469	98	†	0.98
	K053102	1–3	497	94	†	0.93
	K053601	1–5	114	92	†	0.92
	K053701	1–3	104	95	†	0.90
	K053801	1–3	100	86	†	0.86
	K053901	1–3	96	97	†	0.83
B15	K054001	1–4	431	93	†	0.95
	K054002	1–3	430	88	†	0.79
	K054003	1–3	387	99	†	0.96
	K054004	1–2	399	95	0.80	†
	K054005	1–3	397	94	†	0.89
	K054006	1–3	382	95	†	0.93
	K054007	1–3	371	86	†	0.81
	K054008	1–4	357	66	†	0.58
B20	K092001	1–3	78	92	†	0.94
	K096501	1–3	57	98	†	0.79
	K100001	1–2	74	97	0.91	†
	K100101	1–3	76	96	†	0.95
	K100201	1–3	79	85	†	0.77
	K100301	1–3	79	94	†	0.91
B21	K059001	1–3	87	93	†	0.86
	K059101	1–3	83	90	†	0.85
	K059201	1–4	69	90	†	0.94
	K059301	1–4	317	95	†	0.84
	K059801	1–3	62	87	†	0.81
	K059901	1–4	60	92	†	0.89
	K060001	1–3	65	88	†	0.80
	K060101	1–4	76	83	†	0.77
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment.

Last updated 14 September 2009 (GF)

Printer-friendly Version