NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Geography Assessment: 2001

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Geography Assessment: 2001

Range of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 geography assessment: 2001
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
G3	G013402	1–3	600	90	†	0.90
	G014001	1–3	600	99	†	0.99
	G014201	1–4	600	90	†	0.94
	G014301	1–3	600	89	†	0.93
	G014401	1–3	600	95	†	0.94
G4	G032901	1–3	600	90	†	0.91
	G033301	1–3	600	94	†	0.95
	G033501	1–4	600	87	†	0.93
	G033801	1–3	600	94	†	0.96
	G034101	1–3	600	90	†	0.93
G5	G016201	1–3	600	100	†	1.00
	G016302	1–3	600	100	†	0.99
	G016401	1–3	600	96	†	0.98
	G016502	1–3	600	97	†	0.95
	G016701	1–3	600	94	†	0.96
	G017101	1–4	600	88	†	0.94
G6	G034801	1–4	600	97	†	0.99
	G035001	1–4	600	82	†	0.90
	G035301	1–4	600	91	†	0.97
	G035501	1–4	600	90	†	0.94
G7	G036201	1–3	600	98	†	0.99
	G036501	1–4	600	86	†	0.90
	G036801	1–3	600	93	†	0.96
	G037101	1–3	600	93	†	0.95
G8	G012201	1–3	600	99	†	0.99
	G012503	1–3	600	100	†	1.00
	G012902	1–3	600	99	†	0.99
	G013001	1–4	600	94	†	0.98
	G013201	1–3	600	90	†	0.94
G9	G019002	1–3	600	100	†	1.00
	G019003	1–3	600	96	†	0.97
	G019102	1–3	600	95	†	0.95
	G019202	1–3	600	93	†	0.94
	G019302	1–3	600	98	†	0.98
	G019402	1–3	600	95	†	0.96
	G019901	1–4	600	95	†	0.96
	G020001	1–3	600	99	†	0.98
	G020201	1–3	600	95	†	0.94
	G020302	1–3	600	95	†	0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.

Last updated 11 July 2008 (KL)

Printer-friendly Version

Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.

National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education

U.S. Department of Education
Institute of Education Sciences
National Center for Education Statistics