Skip to main content

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Civics National Assessment, by Block and Item: 2018

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Civics National Assessment, by Block and Item: 2018

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 civics DBA national assessment, by block and item: 2018
BlockItemScore rangeSample sizePercentage of exact agreement
C1P0421711-350083
P0421741-340083
P0421771-350089
P0421811-340081
P0421841-340083
P0412021-340083
C2P0421521-350089
P0495011-350077
P0421581-350091
P0421621-340082
P0514011-340093
C3P0412051-340087
P0413091-340089
P0413151-340082
P0521011-440080
C4P0409031-340063
P0409061-340089
P0409101-340084
P0409131-340093
P0505011-440080
P0538011-340085
C5P0411021-340089
P0411061-440075
P0411111-340088
P0411161-340075
C6P0655011-340081
P0642011-440083
P0680011-340081
P0708011-340081
P0665011-340089
P0676011-240091
P0702011-330082
C7P0662011-440081
P0646011-340081
P0691011-340082
P0671011-330078
P0659011-340083
P0659021-240087
P0637011-230087
P0637021-230084
NOTE: DBA = Digitally based assessment. Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW PDF File .
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment.


Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 civics DBA national assessment, by block and item: 2018
BlockItemScore rangeSample sizeCohen’s KappaIntraclass correlation
C1P0421711-3500.71.81
P0421741-3400.59.67
P0421771-3500.82.90
P0421811-3400.70.81
P0421841-3400.72.82
P0412021-3400.74.85
C2P0421521-3500.83.91
P0495011-3400.62.74
P0421581-3400.85.88
P0421621-3400.69.83
P0514011-3400.89.92
C3P0412051-3400.77.81
P0413091-3400.84.90
P0413151-3400.73.84
P0521011-4400.69.82
C4P0409031-3400.42.58
P0409061-3300.70.82
P0409101-3400.71.77
P0409131-3400.89.94
P0505011-4400.71.85
P0538011-3400.65.71
C5P0411021-3400.79.86
P0411061-4400.64.78
P0411111-3400.74.83
P0411161-3400.57.71
C6P0655011-3400.68.69
P0642011-4400.76.89
P0680011-3400.71.80
P0708011-3400.64.74
P0665011-3400.84.91
P0676011-2400.70†
P0702011-3300.70.81
C7P0662011-4400.60.68
P0646011-3400.64.73
P0691011-3400.72.82
P0671011-3300.65.80
P0659011-3400.74.83
P0659021-2400.55†
P0637011-2300.63†
P0637021-2300.60†
† Not applicable.
The intraclass correlation is not reported for dichotomously scored items.
NOTE: DBA = Digitally based assessment. The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW PDF File .
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment.


Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 civics PBA national assessment, by block and item: 2018
BlockItemScore rangeSample sizePercentage of exact agreement
C3P0421711-350083
P0421741-350087
P0421771-350088
P0421811-350081
P0421841-350088
C4P0538011-340090
P0514011-340090
P0521011-440081
P0541011-340085
C5P0409031-340057
P0409061-340088
P0409101-340085
P0409131-340092
C6P0492011-340079
P0505011-440082
P0495011-340076
P0548011-340098
P0622011-340092
C7P0411021-340089
P0411061-440079
P0411111-340089
P0411161-340085
C8P0412021-340077
P0412051-340085
P0412131-440080
C9P0413071-440084
P0413091-340089
P0413151-340083
C10P0421521-350090
P0421551-450082
P0421581-350088
P0421621-350080
NOTE: PBA = Paper-based assessment. Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW PDF File .
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment.


Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 civics PBA national assessment, by block and item: 2018
BlockItemScore rangeSample sizeCohen’s KappaIntraclass correlation
C3P0421711-3400.74.85
P0421741-3400.55.63
P0421771-3400.80.88
P0421811-3400.65.76
P0421841-3400.75.82
C4P0538011-3400.77.80
P0514011-3400.80.86
P0521011-4400.67.83
P0541011-3400.70.76
C5P0409031-3400.29.42
P0409061-3300.60.80
P0409101-3400.72.77
P0409131-3400.86.91
C6P0492011-3400.66.77
P0505011-4400.72.86
P0495011-3400.58.73
P0548011-3300.96.99
P0622011-3300.82.89
C7P0411021-3400.77.85
P0411061-4400.64.77
P0411111-3400.75.84
P0411161-3300.66.79
C8P0412021-3400.64.78
P0412051-3400.70.76
P0412131-4400.64.80
C9
P0413071-4400.70.85
P0413091-3400.83.91
P0413151-3400.69.79
C10P0421521-3400.84.89
P0421551-4400.73.87
P0421581-3400.78.85
P0421621-3400.55.68
NOTE: PBA = Paper-based assessment. The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. The block naming conventions used in this table are described in the document 2018 Block Naming Conventions in Data Products and TDW PDF File .
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Civics Assessment.


Last updated 20 December 2022 (PG)