- Surveys & Programs
- Data & Tools
- Fast Facts
- News & Events
- Publications & Products
- About Us
Table of Contents | Search Technical Documentation | References
NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2014, Rescored in 2018, Grade 8 Civics National Assessment, by Block and Item: 2018
| Block | Item | Score range | Sample size | Percentage of exact agreement |
|---|---|---|---|---|
| C3 | P042171 | 1-3 | 800 | 84 |
| P042174 | 1-3 | 800 | 83 | |
| P042177 | 1-3 | 800 | 87 | |
| P042181 | 1-3 | 800 | 77 | |
| P042184 | 1-3 | 800 | 80 | |
| C4 | P053801 | 1-3 | 800 | 84 |
| P051401 | 1-3 | 800 | 86 | |
| P052101 | 1-4 | 800 | 76 | |
| P054101 | 1-3 | 800 | 80 | |
| C5 | P040903 | 1-3 | 800 | 60 |
| P040906 | 1-3 | 800 | 87 | |
| P040910 | 1-3 | 800 | 86 | |
| P040913 | 1-3 | 800 | 92 | |
| C6 | P049201 | 1-3 | 800 | 79 |
| P050501 | 1-4 | 800 | 82 | |
| P049501 | 1-3 | 800 | 74 | |
| P054801 | 1-3 | 800 | 95 | |
| P062201 | 1-3 | 800 | 83 | |
| C7 | P041102 | 1-3 | 800 | 89 |
| P041106 | 1-4 | 800 | 65 | |
| P041111 | 1-3 | 800 | 82 | |
| P041116 | 1-3 | 800 | 73 | |
| C8 | P041202 | 1-3 | 800 | 75 |
| P041205 | 1-3 | 800 | 80 | |
| P041213 | 1-4 | 800 | 77 | |
| C9 | P041307 | 1-4 | 800 | 84 |
| P041309 | 1-3 | 800 | 87 | |
| P041315 | 1-3 | 800 | 80 | |
| C10 | P042152 | 1-3 | 800 | 90 |
| P042155 | 1-4 | 800 | 73 | |
| P042158 | 1-3 | 800 | 88 | |
| P042162 | 1-3 | 800 | 79 | |
NOTE: PBA = Paper-based assessment.
Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW  .SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment. | ||||
| Block | Item | Score range | Sample size | Cohen’s Kappa | Intraclass correlation |
|---|---|---|---|---|---|
| C3 | P042171 | 1-3 | 800 | .75 | .85 |
| P042174 | 1-3 | 700 | .50 | .53 | |
| P042177 | 1-3 | 800 | .77 | .87 | |
| P042181 | 1-3 | 700 | .59 | .72 | |
| P042184 | 1-3 | 700 | .63 | .72 | |
| C4 | P053801 | 1-3 | 800 | .62 | .63 |
| P051401 | 1-3 | 700 | .75 | .81 | |
| P052101 | 1-4 | 800 | .59 | .75 | |
| P054101 | 1-3 | 800 | .58 | .67 | |
| C5 | P040903 | 1-3 | 800 | .34 | .52 |
| P040906 | 1-3 | 800 | .64 | .78 | |
| P040910 | 1-3 | 700 | .72 | .77 | |
| P040913 | 1-3 | 700 | .86 | .91 | |
| C6 | P049201 | 1-3 | 800 | .65 | .77 |
| P050501 | 1-4 | 800 | .72 | .85 | |
| P049501 | 1-3 | 800 | .53 | .70 | |
| P054801 | 1-3 | 700 | .91 | .95 | |
| P062201 | 1-3 | 700 | .69 | .76 | |
| C7 | P041102 | 1-3 | 700 | .79 | .84 |
| P041106 | 1-4 | 800 | .44 | .60 | |
| P041111 | 1-3 | 800 | .65 | .77 | |
| P041116 | 1-3 | 700 | .46 | .58 | |
| C8 | P041202 | 1-3 | 800 | .60 | .75 |
| P041205 | 1-3 | 800 | .58 | .68 | |
| P041213 | 1-4 | 700 | .61 | .78 | |
| C9 | P041307 | 1-4 | 700 | .73 | .88 |
| P041309 | 1-3 | 800 | .78 | .88 | |
| P041315 | 1-3 | 700 | .65 | .77 | |
| C10 | P042152 | 1-3 | 800 | .84 | .90 |
| P042155 | 1-4 | 800 | .60 | .81 | |
| P042158 | 1-3 | 800 | .77 | .84 | |
| P042162 | 1-3 | 800 | .63 | .77 | |
| NOTE: PBA = Paper-based assessment.
The
intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW . SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment. | |||||
Last updated 20 December 2022 (PG)
