Table of Contents | Search Technical Documentation | References
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Age 13 Mathematics Long-Term Trend Assessment: 2004
| Block | Item | Range of response codes | Sample size | Percent exact agreement | Cohen's Kappa |
|---|---|---|---|---|---|
| M21 | N292401 | 1–2 | 1,000 | 100 | 0.99 |
| N300201 | 1–2 | 900 | 100 | 0.99 | |
| N300401 | 1–2 | 900 | 99 | 0.99 | |
| N300501 | 1–2 | 900 | 99 | 0.98 | |
| N301501 | 1–2 | 900 | 100 | 0.98 | |
| M22 | N301801 | 1–2 | 1,000 | 100 | 1.00 |
| N302401 | 1–2 | 900 | 100 | 0.99 | |
| N302601 | 1–2 | 800 | 100 | 0.99 | |
| N302901 | 1–2 | 900 | 100 | 0.98 | |
| N303201 | 1–2 | 900 | 100 | 0.99 | |
| M23 | N310701 | 1–2 | 900 | 100 | 0.98 |
| N310702 | 1–2 | 1,000 | 100 | 0.99 | |
| N310703 | 1–2 | 1,000 | 99 | 0.97 | |
| N313101 | 1–2 | 900 | 100 | 0.99 | |
| N313102 | 1–2 | 1,000 | 100 | 0.99 | |
| N313103 | 1–2 | 900 | 100 | 0.99 | |
| N315101 | 1–2 | 900 | 100 | 0.99 | |
| N315501 | 1–2 | 900 | 100 | 0.99 | |
| N315701 | 1–2 | 900 | 99 | 0.98 | |
| M24 | N316601 | 1–2 | 900 | 100 | 0.99 |
| N316602 | 1–2 | 900 | 100 | 0.94 | |
| N316603 | 1–2 | 900 | 100 | 0.98 | |
| N316701 | 1–2 | 900 | 100 | 0.98 | |
| N317101 | 1–2 | 900 | 100 | 0.99 | |
| N317102 | 1–2 | 900 | 99 | 0.98 | |
| N317103 | 1–2 | 900 | 100 | 0.99 | |
| N318801 | 1–2 | 800 | 98 | 0.95 | |
| MX51 | N288101 | 1–2 | 900 | 100 | 1.00 |
| N296401 | 1–2 | 900 | 99 | 0.99 | |
| N296701 | 1–2 | 900 | 99 | 0.98 | |
| N296901 | 1–2 | 900 | 99 | 0.98 | |
| N297001 | 1–2 | 900 | 100 | 1.00 | |
| MX52 | N289901 | 1–2 | 900 | 99 | 0.93 |
| N290101 | 1–2 | 900 | 99 | 0.99 | |
| N298201 | 1–2 | 900 | 100 | 1.00 | |
| N298601 | 1–2 | 900 | 100 | 0.98 | |
| N299001 | 1–2 | 900 | 100 | 0.99 | |
| NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment. |
|||||
Last updated 06 April 2009 (GF)