NAEP Analysis and Scaling - Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 Reading Combined National and State Assessments, by Item and Block: 20

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Reading Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 reading combined national and state assessments, by item and block: 2011
Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R059604	1-2	1,300	92
	R059605	1-3	1,300	78
	R059606	1-3	1,300	83
	R059607	1-3	1,300	89
	R059610	1-4	1,300	83
R2	R059803	1-4	1,300	74
	R059806	1-3	1,300	85
	R059807	1-3	1,300	90
	R059809	1-2	1,300	92
R3	R060103	1-2	1,300	86
	R060105	1-2	1,300	92
	R060108	1-4	1,300	80
R4	R060301	1-2	1,300	92
	R060305	1-3	1,200	87
	R060306	1-4	1,300	83
	R060309	1-3	1,300	83
R5	R058504	1-3	1,300	77
	R058508	1-4	1,300	71
	R058509	1-2	1,300	93
R6	R058602	1-3	1,300	81
	R058606	1-3	1,300	84
	R058608	1-4	1,300	75
R7	R058805	1-3	1,300	83
	R058807	1-4	1,300	80
	R058810	1-3	1,300	82
R8	R059106	1-4	1,300	67
	R059108	1-3	1,300	80
	R059109	1-3	1,300	76
	R059110	1-2	1,300	92
R9	R060503	1-3	1,300	89
	R060505	1-4	1,300	83
	R060507	1-3	1,300	79
	R060508	1-3	1,300	81
	R060509	1-3	1,300	86
R10	R060603	1-3	1,300	76
	R060605	1-3	1,200	86
	R060606	1-4	1,300	77
	R060607	1-3	1,300	86
	R060608	1-3	1,300	90
R11	R060803	1-3	1,300	72
	R060807	1-2	1,300	96
	R060808	1-4	1,300	80
	R060810	1-3	1,300	82
	R060811	1-3	1,300	86
R12	R061003	1-3	1,300	83
	R061004	1-2	1,300	96
	R061006	1-2	1,200	90
	R061007	1-4	1,200	73
	R061010	1-3	1,300	78
R13	R061301	1-3	1,300	89
	R061303	1-3	1,200	77
	R061304	1-2	1,300	94
	R061307	1-4	1,300	73
	R061309	1-3	1,300	77

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 reading combined national and state assessments, by item and block: 2011
Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R059604	1-2	1,200	0.72	†
	R059605	1-3	1,100	0.58	0.70
	R059606	1-3	1,200	0.69	0.79
	R059607	1-3	1,000	0.79	0.88
	R059610	1-4	1,100	0.74	0.86
R2	R059803	1-4	1,200	0.61	0.77
	R059806	1-3	1,100	0.68	0.78
	R059807	1-3	1,100	0.80	0.84
	R059809	1-2	1,100	0.82	†
R3	R060103	1-2	1,200	0.69	†
	R060105	1-2	1,200	0.84	†
	R060108	1-4	1,200	0.70	0.79
R4	R060301	1-2	1,200	0.78	†
	R060305	1-3	1,200	0.72	0.76
	R060306	1-4	1,200	0.75	0.86
	R060309	1-3	1,100	0.69	0.76
R5	R058504	1-3	1,200	0.58	0.65
	R058508	1-4	1,200	0.54	0.75
	R058509	1-2	1,100	0.84	†
R6	R058602	1-3	1,200	0.68	0.73
	R058606	1-3	1,200	0.74	0.80
	R058608	1-4	1,200	0.64	0.82
R7	R058805	1-3	1,200	0.70	0.78
	R058807	1-4	1,200	0.65	0.78
	R058810	1-3	1,200	0.69	0.72
R8	R059106	1-4	1,200	0.49	0.69
	R059108	1-3	1,200	0.68	0.81
	R059109	1-3	1,200	0.63	0.74
	R059110	1-2	1,200	0.79	†
R9	R060503	1-3	1,200	0.83	0.91
	R060505	1-4	1,200	0.74	0.90
	R060507	1-3	1,200	0.66	0.58
	R060508	1-3	1,200	0.66	0.74
	R060509	1-3	1,100	0.76	0.85
R10	R060603	1-3	1,200	0.62	0.76
	R060605	1-3	1,200	0.73	0.79
	R060606	1-4	1,200	0.54	0.64
	R060607	1-3	1,200	0.70	0.75
	R060608	1-3	1,200	0.85	0.90
R11	R060803	1-3	1,200	0.51	0.63
	R060807	1-2	1,200	0.90	†
	R060808	1-4	1,100	0.69	0.85
	R060810	1-3	1,000	0.63	0.76
	R060811	1-3	900	0.70	0.80
R12	R061003	1-3	1,200	0.72	0.83
	R061004	1-2	1,200	0.82	†
	R061006	1-2	1,200	0.79	†
	R061007	1-4	1,000	0.50	0.67
	R061010	1-3	1,000	0.50	0.59
R13	R061301	1-3	1,200	0.80	0.84
	R061303	1-3	1,200	0.50	0.59
	R061304	1-2	1,200	0.88	†
	R061307	1-4	1,200	0.61	0.82
	R061309	1-3	1,100	0.59	0.74

Last updated 06 January 2016 (GF)

Printer-friendly Version