NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Science Assessment: 2000

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Science Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 science assessment: 2000
Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
S3	K040601	1?3	1,100	99	?	1.00
	K040603	1?3	1,100	97	?	0.98
	K040604	1?3	1,100	95	?	0.97
	K040605	1?3	1,100	96	?	0.97
	K040606	1?4	1,100	97	?	0.99
	K040607	1?4	1,100	96	?	0.98
	K040608	1?4	1,100	95	?	0.98
	K040609	1?4	1,100	97	?	0.99
	K040610	1?4	1,100	96	?	0.98
S5	K040801	1?3	1,100	97	?	0.98
	K040802	1?3	1,100	98	?	0.97
	K040803	1?3	1,100	97	?	0.96
	K040804	1?3	1,100	94	?	0.94
	K040805	1?3	1,100	94	?	0.92
	K040806	1?2	1,100	94	0.90	?
	K040808	1?3	1,100	98	?	0.99
	K040809	1?3	1,100	96	?	0.97
S6	K031301	1?4	1,100	95	?	0.89
	K031302	1?3	1,100	91	?	0.85
	K031305	1?3	1,100	94	?	0.86
	K031306	1?3	1,100	97	?	0.85
	K031307	1?3	1,100	93	?	0.94
	K031308	1?3	1,100	99	?	0.99
	K031309	1?4	1,100	96	?	0.94
S7	K097901	1?3	1,500	98	?	0.99
S7	K098501	1?3	1,500	97	?	0.97
S8	K041306	1?3	1,500	92	?	0.94
	K041307	1?3	1,500	94	?	0.94
	K041401	1?3	1,500	97	?	0.97
	K041402	1?3	1,500	99	?	0.98
	K041403	1?3	1,500	91	?	0.87
S9	K031602	1?3	1,500	97	?	0.98
	K031603	1?3	1,500	99	?	0.99
	K031604	1?3	1,500	99	?	0.99
	K031606	1?3	1,500	96	?	0.96
	K031607	1?4	1,500	93	?	0.95
	K031608	1?3	1,500	95	?	0.94
	K031609	1?3	1,500	95	?	0.96
	K031610	1?3	1,500	97	?	0.96
	K031611	1?3	1,500	99	?	0.98
	K031613	1?3	1,500	98	?	0.95
S10	K092601	1?3	1,100	97	?	0.98
	K093601	1?3	1,100	99	?	0.99
	K095801	1?2	1,100	93	0.86	?
	K095901	1?3	1,100	95	?	0.96
	K096101	1?3	1,100	96	?	0.96
	K099001	1?3	1,100	98	?	0.98
S11	K043001	1?3	1,100	97	?	0.96
	K043101	1?3	1,100	94	?	0.95
	K043102	1?4	1,100	95	?	0.98
	K043103	1?3	1,100	95	?	0.88
	K043501	1?3	1,100	96	?	0.96
	K043601	1?3	1,100	96	?	0.97
	K043602	1?3	1,100	98	?	0.97
	K043603	1?3	1,100	95	?	0.97
S12	K047201	1?4	1,100	96	?	0.98
	K047301	1?3	1,100	99	?	0.99
	K047401	1?3	1,100	96	?	0.98
	K047901	1?3	1,100	99	?	0.99
	K048001	1?3	1,100	100	?	1.00
	K048101	1?4	1,100	95	?	0.95
	K048102	1?3	1,100	98	?	0.97
	K048103	1?2	1,100	97	0.95	?
S13	K048601	1?3	1,100	98	?	0.98
	K048901	1?3	1,100	99	?	0.99
	K049001	1?3	1,100	99	?	0.91
	K049301	1?3	1,100	100	?	1.00
	K049401	1?3	1,100	97	?	0.97
	K049402	1?3	1,100	95	?	0.97
	K049403	1?3	1,100	94	?	0.95
	K049404	1?4	1,100	89	?	0.90
S14	K035601	1?3	1,500	91	?	0.91
	K035801	1?3	1,500	96	?	0.97
	K035901	1?3	1,500	96	?	0.97
	K036101	1?3	1,500	96	?	0.97
	K036301	1?3	1,500	95	?	0.97
	K036401	1?3	1,500	98	?	0.98
	K036402	1?3	1,500	96	?	0.96
	K036403	1?3	1,500	96	?	0.94
	K036404	1?3	1,500	95	?	0.88
	K036701	1?3	1,500	99	?	0.98
	K036801	1?3	1,500	96	?	0.96
S15	K037301	1?3	1,100	97	?	0.96
	K037401	1?3	1,100	92	?	0.91
	K037501	1?3	1,100	95	?	0.88
	K037601	1?3	1,100	95	?	0.97
	K037701	1?4	1,100	99	?	0.99
	K037703	1?3	1,100	97	?	0.97
	K038101	1?2	1,100	98	0.94	?
	K038201	1?3	1,100	95	?	0.91
	K038301	1?5	1,100	91	?	0.96
S20	K092801	1?3	1,100	99	?	0.99
	K093701	1?3	1,100	96	?	0.97
	K093901	1?3	1,100	94	?	0.95
	K094901	1?3	1,100	96	?	0.93
	K095401	1?3	1,100	95	?	0.91
	K097001	1?3	1,100	94	?	0.91
S21	K045301	1?3	1,100	97	?	0.98
	K045601	1?4	1,100	97	?	0.96
	K045701	1?3	1,100	94	?	0.94
	K045801	1?3	1,100	96	?	0.95
	K046301	1?3	1,100	98	?	0.97
	K046401	1?3	1,100	94	?	0.89
	K046501	1?3	1,100	98	?	0.98
	K046601	1?3	1,100	95	?	0.97
	K046701	1?4	1,100	92	?	0.94
? The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Last updated 15 July 2008 (KL)

Printer-friendly Version