NAEP Analysis and Scaling - Range of Response Codes, Percentage Exact Agreement, and Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2002, by Block and Item, Grade 8 Writing Combined National and

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2002, by Block and Item, Grade 8 Writing Combined National and State Main Assessment: 2002

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and intraclass correlation for the constructed-response items from previous assessment years, rescored in 2002, by block and item, grade 8 writing combined national and state main assessment: 2002
Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
W3	W006002	1–6	2,800	66	†	0.91
W4	W006102	1–6	3,400	72	†	0.94
W5	W006202	1–6	3,200	73	†	0.94
W6	W006302	1–6	3,500	68	†	0.91
W7	W006402	1–6	3,000	75	†	0.95
W8	W006502	1–6	3,800	70	†	0.94
W11	W006802	1–6	2,800	69	†	0.94
W12	W006902	1–6	3,300	68	†	0.94
W13	W007002	1–6	2,000	71	†	0.95
W14	W007102	1–6	3,500	72	†	0.95
W16	W007302	1–6	3,500	69	†	0.94
W17	W007402	1–6	2,600	71	†	0.94
W20	W007702	1–6	1,600	75	†	0.96
W21	W007802	1–6	3,200	68	†	0.93
W22	W007902	1–6	2,600	75	†	0.95
W23	W008002	1–6	4,600	74	†	0.96
W24	W008102	1–6	3,700	71	†	0.94
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Writing Assessment.

Last updated 26 March 2009 (GF)

Printer-friendly Version