NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Science Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 12 Science Assessment: 2000

Range of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 science assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
S3	K049501	1?4	600	99	?	0.99
	K049502	1?5	600	93	?	0.96
	K049503	1?3	600	92	?	0.95
	K049504	1?3	600	93	?	0.96
	K049505	1?3	600	95	?	0.98
	K049506	1?3	600	93	?	0.92
S5	K040801	1?3	600	98	?	0.99
	K040802	1?3	600	98	?	0.98
	K040803	1?3	600	98	?	0.97
	K040804	1?3	600	92	?	0.93
	K040805	1?3	600	95	?	0.96
	K040806	1?2	600	92	0.88	?
	K040808	1?3	600	98	?	0.99
	K040809	1?3	600	97	?	0.99
S6	K049701	1?3	700	99	?	0.99
	K049702	1?3	700	98	?	0.99
	K049703	1?3	700	97	?	0.97
	K049704	1?3	700	97	?	0.98
	K049705	1?4	700	92	?	0.96
	K049706	1?3	700	92	?	0.96
	K049707	1?5	700	92	?	0.98
	K049708	1?2	700	98	0.97	?
S8	K041306	1?3	800	94	?	0.96
	K041307	1?3	800	93	?	0.95
	K041401	1?3	800	98	?	0.98
	K041402	1?3	800	98	?	0.98
	K041403	1?3	800	91	?	0.90
	K041404	1?3	800	97	?	0.98
	K041406	1?3	800	100	?	1.00
S9	K049901	1?3	900	91	?	0.94
	K049902	1?3	900	89	?	0.93
	K049903	1?3	900	94	?	0.96
	K049904	1?3	900	97	?	0.98
	K049907	1?3	900	94	?	0.97
	K049908	1?3	900	97	?	0.98
	K049909	1?4	900	99	?	1.00
	K049911	1?3	900	95	?	0.95
	K049912	1?5	900	88	?	0.96
	K049914	1?3	900	98	?	0.99
S10	K090301	1?3	600	99	?	0.99
	K090501	1?3	600	99	?	0.98
	K092101	1?3	600	99	?	0.99
	K092701	1?3	600	95	?	0.95
	K094601	1?3	600	97	?	0.95
	K098601	1?3	600	92	?	0.89
S11	K051701	1?2	600	97	0.95	?
	K051801	1?3	600	92	?	0.94
	K052301	1?3	600	94	?	0.94
	K052401	1?4	600	92	?	0.97
	K052402	1?3	600	98	?	0.98
	K052501	1?4	600	98	?	0.99
	K052502	1?4	600	98	?	0.99
	K052503	1?3	600	98	?	0.99
S12	K047201	1?4	600	96	?	0.99
	K047301	1?3	600	97	?	0.98
	K047401	1?3	600	95	?	0.97
	K047901	1?3	600	99	?	0.99
	K048001	1?3	600	99	?	0.99
	K048101	1?4	600	87	?	0.92
	K048102	1?3	600	95	?	0.90
	K048103	1?2	600	95	0.91	?
S13	K048601	1?3	600	96	?	0.96
	K048901	1?3	600	99	?	1.00
	K049001	1?3	600	97	?	0.89
	K049301	1?3	700	99	?	0.98
	K049401	1?3	600	97	?	0.96
	K049402	1?3	700	93	?	0.96
	K049403	1?3	600	93	?	0.94
	K049404	1?4	600	88	?	0.91
S14	K052901	1?4	900	99	?	1.00
	K053001	1?3	900	98	?	0.98
	K053101	1?3	900	99	?	0.99
	K053102	1?3	900	100	?	1.00
	K053601	1?5	900	96	?	0.97
	K053701	1?3	900	99	?	0.99
	K053801	1?3	900	94	?	0.95
	K053901	1?3	900	98	?	0.97
S15	K054001	1?4	600	98	?	0.99
	K054002	1?3	600	98	?	0.98
	K054003	1?3	600	100	?	1.00
	K054004	1?2	600	100	1.00	?
	K054005	1?3	600	100	?	1.00
	K054006	1?3	600	98	?	0.98
	K054007	1?3	600	100	?	1.00
	K054008	1?4	600	96	?	0.98
S20	K092001	1?3	600	94	?	0.97
S20	K096501	1?3	600	100	?	1.00
S21	K059001	1?3	600	96	?	0.97
	K059101	1?3	600	93	?	0.91
	K059201	1?4	600	96	?	0.98
	K059301	1?4	600	99	?	0.99
	K059801	1?3	600	97	?	0.97
	K059901	1?4	600	99	?	0.99
	K060001	1?3	600	94	?	0.97
	K060101	1?4	600	94	?	0.96
? The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version

Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.

National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education

U.S. Department of Education
Institute of Education Sciences
National Center for Education Statistics