Table of Contents | Search Technical Documentation | References
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Age 9 Mathematics Long-Term Trend Bridge Study: 2004
| Block | Item | Range of response codes | Sample size | Percent exact agreement | Cohen's Kappa |
|---|---|---|---|---|---|
| M1 | N270001 | 1–2 | 600 | 99 | 0.98 |
| N270901 | 1–2 | 600 | 99 | 0.90 | |
| N276101 | 1–2 | 600 | 99 | 0.98 | |
| N276801 | 1–2 | 700 | 100 | 0.92 | |
| N276802 | 1–2 | 600 | 99 | 0.96 | |
| N276803 | 1–2 | 600 | 99 | 0.97 | |
| N284001 | 1–2 | 600 | 99 | 0.96 | |
| N284002 | 1–2 | 500 | 99 | 0.97 | |
| N286101 | 1–2 | 600 | 99 | 0.97 | |
| M2 | N271101 | 1–2 | 600 | 99 | 0.96 |
| N275401 | 1–2 | 600 | 99 | 0.97 | |
| N276001 | 1–2 | 600 | 99 | 0.96 | |
| N276002 | 1–2 | 500 | 100 | 0.99 | |
| N277501 | 1–2 | 600 | 99 | 0.98 | |
| N277601 | 1–2 | 600 | 100 | 0.99 | |
| N277602 | 1–2 | 600 | 99 | 0.98 | |
| N277603 | 1–2 | 600 | 99 | 0.99 | |
| N286102 | 1–2 | 500 | 98 | 0.96 | |
| M21 | N292101 | 1–2 | 1,200 | 100 | 0.99 |
| N292401 | 1–2 | 1,200 | 100 | 0.99 | |
| N293001 | 1–2 | 1,200 | 100 | 0.99 | |
| N293301 | 1–2 | 1,200 | 100 | 0.99 | |
| N293501 | 1–2 | 1,200 | 100 | 0.99 | |
| M22 | N294201 | 1–2 | 600 | 99 | 0.89 |
| N294301 | 1–2 | 600 | 100 | 1.00 | |
| N295301 | 1–2 | 600 | 100 | 0.99 | |
| N295401 | 1–2 | 600 | 99 | 0.99 | |
| N295701 | 1–2 | 400 | 100 | 1.00 | |
| NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment. |
|||||
Last updated 06 April 2009 (GF)