Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Science National Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Science National Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 12 science national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
S3	K133301	1-5	500	93
	K157201	1-3	500	83
	K157202	1-3	500	92
	K139001	1-3	500	85
	K126303	1-5	500	94
	K126304	1-3	500	90
	K126202	1-3	500	95
S4	K133801	1-5	500	77
	K128403	1-4	500	85
	K137201	1-4	500	93
	K157901	1-3	500	88
	K144401	1-4	500	89
S5	K129601	1-3	500	86
	K143903	1-3	500	93
	K131001	1-4	500	90
	K130301	1-3	500	85
	K158001	1-4	500	92
S6	K135001	1-4	500	87
	K158201	1-3	500	84
	K158202	1-3	500	96
	K158203	1-3	500	89
	K158204	1-3	500	92
	K139801	1-3	500	82
	K143801	1-3	500	96
	K134503	1-3	500	84
S7	K136101	1-4	500	89
	K136102	1-3	500	92
	K136103	1-4	500	92
	K132701	1-3	500	94
	K135401	1-3	500	94
	K158301	1-3	500	87
S8	K127201	1-5	500	65
	K137102	1-4	500	93
	K136401	1-3	500	87
	K142401	1-3	500	95
	K158401	1-3	500	91
	K158402	1-3	500	94
	K137651	1-5	500	96
S9	K176601	1-3	500	94
	K177601	1-3	500	93
	K177602	1-3	500	93
	K173301	1-3	500	97
S10	K172901	1-4	500	96
	K179701	1-3	500	87
	K172501	1-4	500	86
	K174701	1-3	500	77
S11	K127701	1-4	500	86
	K125802	1-3	500	88
	K125803	1-3	500	97
	K144202	1-3	500	85
	K141701	1-4	500	90
S12	K129001	1-3	500	93
	K128701	1-5	500	75
	K133401	1-3	500	85
	K130001	1-4	500	92
	K130002	1-4	500	93
S13	K126001	1-5	500	74
	K140301	1-3	500	89
	K158701	1-3	500	95
	K158702	1-3	500	92
	K140303	1-3	500	88
	K138501	1-4	500	98
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 12 science national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
S3	K133301	1-5	500	.79	.90
	K157201	1-3	500	.73	.84
	K157202	1-3	400	.84	.88
	K139001	1-3	400	.67	.75
	K126303	1-5	500	.87	.97
	K126304	1-3	500	.80	.86
	K126202	1-3	500	.86	.93
S4	K133801	1-5	500	.62	.86
	K128403	1-4	400	.66	.73
	K137201	1-4	500	.87	.96
	K157901	1-3	400	.80	.88
	K144401	1-4	500	.79	.86
S5	K129601	1-3	500	.71	.83
	K143903	1-3	400	.81	.89
	K131001	1-4	500	.76	.78
	K130301	1-3	400	.73	.78
	K158001	1-4	500	.88	.91
S6	K135001	1-4	400	.58	.77
	K158201	1-3	500	.74	.86
	K158202	1-3	400	.89	.95
	K158203	1-3	400	.73	.86
	K158204	1-3	400	.74	.86
	K139801	1-3	500	.71	.79
	K143801	1-3	400	.81	.79
	K134503	1-3	400	.66	.76
S7	K136101	1-4	500	.81	.89
	K136102	1-3	500	.86	.90
	K136103	1-4	500	.84	.89
	K132701	1-3	400	.77	.85
	K135401	1-3	400	.89	.94
	K158301	1-3	400	.71	.77
S8	K127201	1-5	500	.51	.79
	K137102	1-4	400	.87	.97
	K136401	1-3	400	.70	.84
	K142401	1-3	500	.92	.94
	K158401	1-3	400	.79	.87
	K158402	1-3	300	.80	.89
	K137651	1-5	400	.94	.99
S9	K176601	1-3	400	.87	.92
	K177601	1-3	500	.88	.94
	K177602	1-3	500	.88	.95
	K173301	1-3	400	.93	.95
S10	K172901	1-4	500	.94	.98
	K179701	1-3	500	.70	.70
	K172501	1-4	500	.81	.92
	K174701	1-3	500	.60	.75
S11	K127701	1-4	400	.78	.90
	K125802	1-3	400	.70	.75
	K125803	1-3	400	.93	.95
	K144202	1-3	400	.64	.75
	K141701	1-4	500	.86	.93
S12	K129001	1-3	500	.64	.65
	K128701	1-5	500	.64	.88
	K133401	1-3	400	.71	.78
	K130001	1-4	500	.87	.90
	K130002	1-4	500	.90	.94
S13	K126001	1-5	500	.51	.82
	K140301	1-3	500	.79	.87
	K158701	1-3	500	.91	.92
	K158702	1-3	500	.87	.93
	K140303	1-3	400	.80	.89
	K138501	1-4	300	.92	.99
NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version