Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Mathematics National Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Mathematics National Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 12 mathematics national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
M1	M183601	1-3	500	99
	M178001	1-3	500	95
	M183901	1-2	500	99
	M186701	1-3	500	94
	M186702	1-3	500	91
	M175901	1-3	500	98
M2	M306701	1-3	500	92
	M302901	1-3	500	96
	M305901	1-2	500	99
	M305902	1-2	500	99
	M305903	1-2	500	84
	M307501	1-3	500	95
M3	M197303	1-2	500	99
	M197304	1-2	500	100
	M193401	1-3	500	91
	M197501	1-3	500	93
	M197502	1-3	500	95
	M197503	1-3	500	97
	M179201	1-3	500	97
M4	M184201	1-2	500	100
	M184202	1-2	500	98
	M184203	1-2	500	99
	M184204	1-2	500	98
	M184205	1-2	500	99
	M184206	1-2	500	99
	M184207	1-3	500	92
	M197601	1-3	500	95
	M174801	1-3	500	98
M5	M177301	1-3	500	88
	M185703	1-5	500	99
	M187701	1-3	500	99
	M185201	1-2	500	98
M6	M183401	1-3	500	97
	M175601	1-3	500	95
	M175002	1-5	500	99
	M177202	1-2	500	97
M7	M310401	1-2	500	99
	M304001	1-3	500	96
	M310001	1-4	500	95
M8	M193101	1-2	500	100
	M188201	1-5	500	98
	M188001	1-3	500	96
	M194301	1-2	500	100
	M194302	1-2	500	100
	M189001	1-3	500	92
M9	M198701	1-2	500	100
	M191901	1-5	500	96
	M198901	1-3	500	93
M10	M178501	1-3	500	99
	M191701	1-3	500	100
	M199201	1-3	500	100
	M190801	1-3	500	97
	M189701	1-3	500	98
	M189702	1-2	500	96
	M189703	1-2	500	100
M11	M181201	1-2	500	99
	M181202	1-3	500	99
	M181203	1-2	500	99
	M176501	1-3	500	96
	M176502	1-2	500	96
	M176503	1-2	500	97
	M176504	1-2	500	95
	M183101	1-2	500	99
M12	M186001	1-2	500	99
	M181801	1-2	500	99
	M194001	1-3	500	98
	M178701	1-4	500	98
M13	M302701	1-3	500	100
	M305501	1-2	500	100
	M306501	1-2	500	100
	M302101	1-4	500	98
M14	M308001	1-2	500	99
	M310501	1-3	500	100
	M311101	1-3	500	98
	M311102	1-3	500	97
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Mathematics Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 12 mathematics national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
M1	M183601	1-3	400	.98	.98
	M178001	1-3	400	.90	.94
	M183901	1-2	400	.98	†
	M186701	1-3	400	.90	.94
	M186702	1-3	400	.83	.90
	M175901	1-3	400	.87	.90
M2	M306701	1-3	500	.87	.93
	M302901	1-3	400	.96	.98
	M305901	1-2	400	.98	†
	M305902	1-2	400	.95	†
	M305903	1-2	300	.33	†
	M307501	1-3	400	.88	.91
M3	M197303	1-2	400	.96	†
	M197304	1-2	400	1.00	†
	M193401	1-3	500	.74	.87
	M197501	1-3	400	.86	.90
	M197502	1-3	400	.81	.82
	M197503	1-3	300	.83	.92
	M179201	1-3	300	.91	.94
M4	M184201	1-2	400	1.00	†
	M184202	1-2	500	.90	†
	M184203	1-2	400	.99	†
	M184204	1-2	500	.96	†
	M184205	1-2	400	.98	†
	M184206	1-2	400	.96	†
	M184207	1-3	400	.72	.80
	M197601	1-3	400	.91	.94
	M174801	1-3	300	.90	.94
M5	M177301	1-3	500	.78	.90
	M185703	1-5	400	.99	.99
	M187701	1-3	400	.99	1.00
	M185201	1-2	400	.97	†
M6	M183401	1-3	300	.88	.87
	M175601	1-3	500	.91	.94
	M175002	1-5	500	.98	.99
	M177202	1-2	400	.93	†
M7	M310401	1-2	500	.98	†
	M304001	1-3	500	.91	.93
	M310001	1-4	400	.93	.98
M8	M193101	1-2	500	.99	†
	M188201	1-5	400	.96	.99
	M188001	1-3	400	.94	.98
	M194301	1-2	400	1.00	†
	M194302	1-2	400	1.00	†
	M189001	1-3	400	.81	.79
M9	M198701	1-2	500	1.00	†
	M191901	1-5	400	.92	.99
	M198901	1-3	400	.84	.88
M10	M178501	1-3	400	.97	.98
	M191701	1-3	500	1.00	1.00
	M199201	1-3	400	1.00	1.00
	M190801	1-3	400	.95	.96
	M189701	1-3	400	.91	.93
	M189702	1-2	300	.87	†
	M189703	1-2	300	.99	†
M11	M181201	1-2	500	.98	†
	M181202	1-3	500	.98	.98
	M181203	1-2	400	.97	†
	M176501	1-3	400	.94	.97
	M176502	1-2	400	.91	†
	M176503	1-2	400	.88	†
	M176504	1-2	400	.88	†
	M183101	1-2	300	.97	†
M12	M186001	1-2	500	1.00	†
	M181801	1-2	400	1.00	†
	M194001	1-3	400	.97	.98
	M178701	1-4	300	.97	.98
M13	M302701	1-3	500	1.00	1.00
	M305501	1-2	500	1.00	†
	M306501	1-2	500	1.00	†
	M302101	1-4	400	.89	.95
M14	M308001	1-2	500	.99	†
	M310501	1-3	500	.99	.99
	M311101	1-3	400	.95	.99
	M311102	1-3	400	.77	.80
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Mathematics Assessment.

Last updated 09 November 2021 (ML)

Printer-friendly Version