NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Main Assessment, by Item and Block

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessments, by item and block: 2011
Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M149901	1-3	1,800	99
	M150801	1-3	1,800	98
	M150901	1-2	1,800	99
	M151201	1-2	1,800	99
M4	M151501	1-3	1,800	92
	M151601	1-3	1,800	96
	M152602	1-5	1,800	97
M5	M221203	1-3	1,800	99
	M221204	1-3	1,800	87
	M221701	1-2	1,800	99
	M222301	1-3	1,800	99
M6	M141351	1-3	1,900	93
	M141601	1-2	1,900	90
	M141901	1-5	1,900	90
M7	M165301	1-3	1,800	95
	M167001	1-2	1,800	93
	M168501	1-2	1,800	99
	M168502	1-2	1,800	98
	M168503	1-3	1,800	95
	M168701	1-3	1,800	97
	M170101	1-3	1,800	98
M8	M164601	1-3	1,900	98
	M166301	1-3	1,900	93
	M169901	1-2	1,900	98
	M169902	1-2	1,900	98
	M169903	1-3	1,900	94
	M169904	1-2	1,900	94
	M170501	1-3	1,900	89
M9	M165001	1-2	1,800	97
	M165501	1-3	1,800	99
	M165701	1-3	1,800	99
	M165702	1-3	1,800	95
	M168901	1-2	1,800	99
	M170801	1-3	1,800	99
M10	M222701	1-3	1,800	99
	M223101	1-3	1,800	100
	M223301	1-3	1,800	95
	M223801	1-3	1,800	99
M11	M104901	1-3	1,900	97
	M162901	1-3	1,900	95
	M163101	1-3	1,900	93
	M167901	1-3	1,900	96
	M168301	1-3	1,900	95
M12	M153201	1-3	1,900	93
	M153601	1-2	1,900	98
	M153901	1-3	1,900	89

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 4 mathematics combined national and state assessments, by item and block: 2011
Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M149901	1-3	1,700	0.99	0.99
	M150801	1-3	1,600	0.96	0.99
	M150901	1-2	1,700	0.99	†
	M151201	1-2	1,600	0.99	†
M4	M151501	1-3	1,700	0.88	0.94
	M151601	1-3	1,600	0.88	0.95
	M152602	1-5	1,600	0.96	0.97
M5	M221203	1-3	1,700	0.98	0.98
	M221204	1-3	1,700	0.75	0.79
	M221701	1-2	1,700	0.99	†
	M222301	1-3	1,400	0.99	1.00
M6	M141351	1-3	1,700	0.91	0.96
	M141601	1-2	1,600	0.78	†
	M141901	1-5	1,600	0.86	0.94
M7	M165301	1-3	1,700	0.92	0.96
	M167001	1-2	1,600	0.70	†
	M168501	1-2	1,500	0.98	†
	M168502	1-2	1,500	0.96	†
	M168503	1-3	1,300	0.77	0.87
	M168701	1-3	1,700	0.95	0.98
	M170101	1-3	1,700	0.98	0.99
M8	M164601	1-3	1,800	0.99	0.99
	M166301	1-3	1,800	0.90	0.96
	M169901	1-2	1,700	0.94	†
	M169902	1-2	1,700	0.97	†
	M169903	1-3	1,600	0.91	0.96
	M169904	1-2	1,500	0.82	†
	M170501	1-3	1,700	0.83	0.90
M9	M165001	1-2	1,600	0.92	†
	M165501	1-3	1,800	0.99	0.99
	M165701	1-3	1,600	0.98	0.98
	M165702	1-3	1,600	0.92	0.96
	M168901	1-2	1,800	0.98	†
	M170801	1-3	1,600	0.91	0.94
M10	M222701	1-3	1,700	0.98	0.99
	M223101	1-3	1,700	1.00	1.00
	M223301	1-3	1,700	0.91	0.94
	M223801	1-3	1,500	0.99	0.99
M11	M104901	1-3	1,600	0.95	0.96
	M162901	1-3	1,700	0.94	0.96
	M163101	1-3	1,700	0.91	0.94
	M167901	1-3	1,700	0.90	0.96
	M168301	1-3	1,600	0.94	0.97
M12	M153201	1-3	1,800	0.90	0.93
	M153601	1-2	1,800	0.98	†
	M153901	1-3	1,700	0.83	0.91

Last updated 06 January 2016 (GF)

Printer-friendly Version