NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2004, by Block and Item, Age 9 Mathematics Long-Term Tren

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2004, by Block and Item, Age 9 Mathematics Long-Term Trend Assessment: 2004

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2004, by block and item, age 9 mathematics long-term trend assessment: 2004
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B7	K104501	1–3	100	93	†	0.86
	K104601	1–3	100	68	†	0.41
	K105001	1–3	100	91	†	0.90
	K105501	1–3	100	100	†	1.00
	K105601	1–2	100	94	0.84	†
	K106101	1–3	100	97	†	0.48
B8	K041306	1–3	100	79	†	0.82
	K041307	1–3	100	84	†	0.85
	K041401	1–3	100	90	†	0.85
	K041402	1–3	100	89	†	0.85
	K041403	1–3	100	89	†	0.86
	K041404	1–3	100	89	†	0.88
	K041406	1–3	100	96	†	0.88
B9	K049901	1–3	600	92	†	0.92
	K049902	1–3	500	90	†	0.88
	K049903	1–3	500	93	†	0.89
	K049904	1–3	500	94	†	0.92
	K049907	1–3	500	93	†	0.93
	K049908	1–3	500	94	†	0.93
	K049909	1–4	400	98	†	0.96
	K049911	1–3	500	88	†	0.87
	K049912	1–5	400	77	†	0.89
	K049914	1–3	400	97	†	0.97
B10	K090301	1–3	100	92	†	0.92
	K090501	1–3	100	99	†	0.97
	K092101	1–3	100	98	†	0.98
	K092701	1–3	100	94	†	0.93
	K094601	1–3	100	94	†	0.52
	K098601	1–3	100	88	†	0.80
B11	K051701	1–2	400	90	0.80	†
	K051801	1–3	400	80	†	0.80
	K052301	1–3	400	88	†	0.82
	K052401	1–4	300	91	†	0.95
	K052402	1–3	300	95	†	0.91
	K052501	1–4	400	94	†	0.97
	K052502	1–4	400	96	†	0.97
	K052503	1–3	400	92	†	0.91
B12	K047201	1–4	100	85	†	0.91
	K047301	1–3	100	93	†	0.92
	K047401	1–3	100	89	†	0.88
	K047901	1–3	100	87	†	0.90
	K048001	1–3	100	95	†	0.88
	K048101	1–4	400	81	†	0.83
	K048102	1–3	300	93	†	0.64
	K048103	1–2	300	89	0.79	†
B13	K048601	1–3	100	86	†	0.87
	K048901	1–3	100	92	†	0.94
	K049001	1–3	100	99	†	0.96
	K049301	1–3	100	95	†	0.81
	K049401	1–3	100	93	†	0.87
	K049402	1–3	100	79	†	0.78
	K049403	1–3	100	80	†	0.67
	K049404	1–4	100	72	†	0.69
B14	K052901	1–4	100	95	†	0.98
	K053001	1–3	100	91	†	0.84
	K053101	1–3	500	98	†	0.98
	K053102	1–3	500	94	†	0.93
	K053601	1–5	100	92	†	0.92
	K053701	1–3	100	95	†	0.90
	K053801	1–3	100	86	†	0.86
	K053901	1–3	100	97	†	0.83
B15	K054001	1–4	400	93	†	0.95
	K054002	1–3	400	88	†	0.79
	K054003	1–3	400	99	†	0.96
	K054004	1–2	400	95	0.80	†
	K054005	1–3	400	94	†	0.89
	K054006	1–3	400	95	†	0.93
	K054007	1–3	400	86	†	0.81
	K054008	1–4	400	66	†	0.58
B20	K092001	1–3	100	92	†	0.94
	K096501	1–3	100	98	†	0.79
	K100001	1–2	100	97	0.91	†
	K100101	1–3	100	96	†	0.95
	K100201	1–3	100	85	†	0.77
	K100301	1–3	100	94	†	0.91
B21	K059001	1–3	100	93	†	0.86
	K059101	1–3	100	90	†	0.85
	K059201	1–4	100	90	†	0.94
	K059301	1–4	300	95	†	0.84
	K059801	1–3	100	87	†	0.81
	K059901	1–4	100	92	†	0.89
	K060001	1–3	100	88	†	0.80
	K060101	1–4	100	83	†	0.77
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment.

Last updated 06 April 2009 (GF)

Printer-friendly Version