2017 Writing

Reflecting the increasing use of digital devices in writing composition, a new NAEP Writing Framework was developed for a 2011 assessment. The new framework specifies that students’ writing skill be assessed “…using word processing software with commonly available tools (NAEP Writing Framework, p. vi).” To implement this framework, NCES first administered digitally based writing assessments to students in grades 8 and 12 in 2011 and then to students in grade 4, along with grade 8, in 2017.

The 2017 grade 8 writing assessment differed from the 2011 assessment, primarily in the use of digital devices (tablets with attached keyboards in 2017 and laptops in 2011 each with different software). As limited research was available comparing writing performance on digital devices, NCES conducted a comparability study with grade 8 students in 2017. The comparability study was intended to help explore whether changes in the digital device had an effect on students’ writing performance. The study allowed comparisons across years between the operational assessments in 2011 and 2017, as well as across devices and other test administration conditions within 2017.

Those comparisons showed a pattern of lower performance for students assessed in the 2017 operational tablet condition compared to students assessed in either the 2011 or the 2017 laptop condition. Although the comparability study provided valuable insights, results from the study alone were not sufficient for NCES to determine whether the differences in performance could be attributed solely to the change in device or to differences in students’ writing skills and abilities. Additional investigation is necessary to determine what factors might have influenced the 2017 writing performance of grade 8 students.

Unlike at grade 8, a comparability study could not be performed at grade 4 because 2017 marked the first operational digitally based NAEP writing assessment. NCES conducted a pilot study on laptops in 2012 to prepare for the eventual transition to a digitally based writing assessment. Although performance data between the 2012 pilot study and the 2017 assessment showed broadly similar patterns, data collected during the 2017 assessment, such as typing speed, response length, and student-reported familiarity with the various writing devices, revealed concerns, suggesting that students’ writing achievement may have been affected by how the assessment was administered in 2017. NCES concluded that additional analyses and investigations are necessary.

Advisory panels, regularly consulted to assure NAEP’s high standard of reliability and validity before the release of assessment results, agreed with NCES that an in-depth exploration of the 2017 assessment was imperative. NCES also plans to conduct other studies to further enhance understanding of elementary and secondary school students’ writing skill in the context of technology in classrooms.

• NAEP 2017 Writing Assessments

  Read the Technical Summary of Preliminary Analyses of NAEP 2017 Writing Assessments.