Table of Contents | Search Technical Documentation | References
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Mathematics Combined National and State Main Assessment: 2003
| Block | Item | Range of response codes | Sample size | Percent exact agreement | Cohen's Kappa | Intraclass correlation |
|---|---|---|---|---|---|---|
| M3 | M019901 | 1–3 | 2,000 | 98 | † | 0.97 |
| M020101 | 1–2 | 1,800 | 98 | 0.96 | † | |
| M043501 | 1–5 | 1,800 | 93 | † | 0.97 | |
| M047301 | 1–4 | 1,900 | 98 | † | 0.96 | |
| M074301 | 1–2 | 2,000 | 97 | 0.95 | † | |
| M074501 | 1–3 | 1,900 | 88 | † | 0.90 | |
| M074801 | 1–3 | 2,000 | 97 | † | 0.97 | |
| M074901 | 1–3 | 1,900 | 92 | † | 0.94 | |
| M075001 | 1–3 | 2,000 | 98 | † | 0.98 | |
| N277903 | 1–2 | 2,000 | 99 | 0.97 | † | |
| M4 | M046001 | 1–5 | 2,000 | 99 | † | 0.98 |
| M046601 | 1–4 | 2,000 | 95 | † | 0.96 | |
| M046801 | 1–5 | 2,000 | 99 | † | 0.99 | |
| M046901 | 1–5 | 2,000 | 99 | † | 0.99 | |
| M066601 | 1–3 | 2,000 | 91 | † | 0.93 | |
| M068001 | 1–3 | 2,000 | 98 | † | 0.98 | |
| M068002 | 1–3 | 2,000 | 97 | † | 0.98 | |
| M068003 | 1–3 | 1,800 | 98 | † | 0.98 | |
| M068004 | 1–5 | 1,800 | 91 | † | 0.98 | |
| M5 | M019701 | 1–2 | 1,800 | 99 | 0.98 | † |
| M020001 | 1–2 | 1,800 | 99 | 0.98 | † | |
| M066301 | 1–3 | 1,500 | 96 | † | 0.94 | |
| M066501 | 1–3 | 1,700 | 95 | † | 0.98 | |
| M067901 | 1–3 | 1,300 | 95 | † | 0.97 | |
| M085401 | 1–5 | 2,000 | 92 | † | 0.96 | |
| M085701 | 1–3 | 2,000 | 94 | † | 0.96 | |
| M085901 | 1–3 | 2,000 | 95 | † | 0.96 | |
| M6 | M019801 | 1–3 | 1,700 | 94 | † | 0.92 |
| M020201 | 1–2 | 2,000 | 95 | 0.88 | † | |
| M020301 | 1–4 | 2,000 | 98 | † | 0.97 | |
| M020401 | 1–2 | 2,000 | 99 | 0.98 | † | |
| M020501 | 1–2 | 2,000 | 98 | 0.96 | † | |
| M7 | M020701 | 1–4 | 1,300 | 80 | † | 0.73 |
| M072201 | 1–3 | 2,000 | 99 | † | 0.98 | |
| M072202 | 1–3 | 2,000 | 97 | † | 0.98 | |
| M072501 | 1–3 | 1,800 | 90 | † | 0.90 | |
| M072601 | 1–3 | 1,400 | 98 | † | 0.97 | |
| M072701 | 1–5 | 1,300 | 87 | † | 0.95 | |
| M8 | M010631 | 1–3 | 2,000 | 97 | † | 0.97 |
| M040001 | 1–3 | 1,900 | 98 | † | 0.98 | |
| M072401 | 1–3 | 1,500 | 91 | † | 0.94 | |
| M087001 | 1–3 | 2,000 | 97 | † | 0.98 | |
| M087301 | 1–5 | 1,300 | 94 | † | 0.97 | |
| M091201 | 1–3 | 1,700 | 92 | † | 0.94 | |
| M9 | M040201 | 1–2 | 1,800 | 94 | 0.83 | † |
| M043201 | 1–2 | 2,100 | 98 | 0.94 | † | |
| M043301 | 1–3 | 1,900 | 98 | † | 0.98 | |
| M043401 | 1–4 | 2,000 | 96 | † | 0.99 | |
| M043402 | 1–4 | 2,001 | 98 | † | 0.99 | |
| M043403 | 1–3 | 2,000 | 98 | † | 0.97 | |
| M066701 | 1–3 | 2,100 | 94 | † | 0.96 | |
| M066801 | 1–3 | 2,000 | 95 | † | 0.95 | |
| M086601 | 1–3 | 2,000 | 98 | † | 0.97 | |
| M091401 | 1–5 | 1,800 | 93 | † | 0.97 | |
| M10 | M039201 | 1–2 | 2,000 | 99 | 0.98 | † |
| M039301 | 1–3 | 2,000 | 98 | † | 0.98 | |
| M066901 | 1–5 | 2,000 | 85 | † | 0.97 | |
| M074701 | 1–3 | 2,000 | 98 | † | 0.96 | |
| M075101 | 1–5 | 2,000 | 87 | † | 0.96 | |
| M091101 | 1–3 | 2,000 | 95 | † | 0.96 | |
| M11 | M096001 | 1–2 | 2,000 | 95 | 0.92 | † |
| M097201 | 1–3 | 2,000 | 94 | † | 0.97 | |
| M097401 | 1–3 | 1,900 | 99 | † | 0.98 | |
| M12 | M100201 | 1–2 | 2,000 | 100 | 0.99 | † |
| M100501 | 1–2 | 2,000 | 99 | 0.98 | † | |
| M101401 | 1–2 | 2,000 | 97 | 0.94 | † | |
| † The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Mathematics Assessment. |
||||||
Last updated 25 March 2009 (GF)