Frequently Asked Questions for the 2018 NAEP Oral Reading Fluency (ORF) Study

The 2018 NAEP Oral Reading Fluency study Click to open pdf., the first such NAEP study since 2002, was administered to a nationally representative sample of over 1,800 fourth-graders from 180 public schools.

Students first completed the NAEP fourth-grade reading assessment. Then they read four short passages out loud to assess their oral reading fluency (i.e., the ability to read text aloud with speed, accuracy, and proper expression) as well as two kinds of word lists to assess the skills that provide the foundation for oral reading fluency: word recognition and phonological decoding (defined below). Both the NAEP reading tasks and ORF tasks were administered to students digitally, with student responses recorded on tablets.

“Word recognition” is the ability to pronounce familiar words. “Phonological decoding” is the ability to pronounce unfamiliar words based on knowledge of the relationships between letters (or groups of letters) and sounds.

There were several major task and administrative differences between the two assessments. For example, the 2018 ORF study (a) provided more information, such as foundational skills, than the 2002 oral reading study Click to open pdf.; (b) operated in the new eNAEP platform on tablets; (c) was group administered; and (d) did not require students to read the entire passages silently before reading orally. See the ORF website for details.

The ORF study was done to provide an in-depth examination of the difficulties faced by students who perform below NAEP Basic in terms of fluency and foundational skills (word recognition and phonological decoding). In 2019, over a third of fourth-graders in public schools performed below the NAEP Basic achievement level on the NAEP reading assessment.

Oral reading fluency had not been measured since the NAEP oral reading study in 2002, and foundational skills had never been measured in NAEP. Research shows that reading comprehension is critically dependent on oral reading fluency and the foundational skills that underlie it. Students who have these skills are more likely to comprehend connected text because they are able to conserve cognitive resources that can be applied to the comprehension of meaning (see Highlights of the 2018 NAEP Oral Reading Fluency Study Click to open pdf. for citations).

When teachers talk about learning “sight words,” they usually mean the very frequent English words, such as the 220 words on the Dolch Word List (e.g., big, are, yes). They may use flash cards to help children learn to recognize these words rapidly.

In the ORF study, the list of words used to assess word recognition was not limited to very frequent words. Research suggests that fast and accurate word recognition, as conceptualized in the ORF study, should be the goal not only for very frequent “sight words", but also for other frequently occurring words and moderately frequent words, including the first 4,000 words in frequency, which make up about 90% of the words in the fourth-grade NAEP reading assessment.

The ORF tasks were administered in a group with approximately 10 students in the same room, spaced at least 6.5 feet apart. We learned in pilot studies that administering the test to small groups of students sitting 5-8 feet apart with earbuds on did not impact other students’ read-aloud performance or the quality of the audio recordings.

Using the e-NAEP platform (a delivery system on which digitally based NAEP assessments are delivered), students were presented text on a tablet computer provided by NCES. Students’ reading aloud performance was recorded and the recordings were then transferred from the tablet to a NAEP database where a speech analysis system processed the recordings to produce a profile of scores, including total number of words read, number of words read correctly, reading rate, and reading accuracy.

Several cognitive laboratory studies were conducted that included the pseudoword task. The instructions to students were: “Now, you will read out loud from a list of made-up words.” Then after giving some examples, students were told, “Again, don't worry if you are not able to finish reading the whole list. Read as much as you can. If you finish reading the list before time is up, tap the NEXT arrow to go on to the next task. Now it's your turn.”

In the post-assessment interviews conducted during the pilot studies, all of the participating students said they were not bothered by others reading aloud. Similarly, we did not receive any reports from field administrators that students were confused by the pseudoword task. Finally, we found a strong correlation between pseudowords and reading comprehension, which is an indication that the pseudoword measure was valid.

Expression was the only ORF measure that was not scored automatically. It was scored by human judges, not a machine. However, the ORF assessment software is expected to be capable of producing automated passage reading expression scores in the future using the existing scoring rubric. As with all other ORF measures, we will examine the correlation between automated scoring and human ratings to ensure the reliability and validity of automated scoring.

To better understand the students who perform the below NAEP Basic achievement level, we divided these students into three subgroups: below NAEP Basic Low, below NAEP Basic Medium, and below NAEP Basic High.

The study showed that foundational skills (word reading and phonological decoding, defined in the first question in the FAQs) are underdeveloped in students performing below NAEP Basic, particularly for students in the below NAEP Basic Low subgroup.

The study also showed that students in the below NAEP Basic Low subgroup read out loud with a lack of appropriate expression (intonation, rhythm, emphasis, and pausing), which is an indicator of poor comprehension. This makes it difficult for them to engage in the cognitive processes described in the 2017 NAEP reading framework.

Approximately 51 percent of all Black fourth-graders and 46 percent of all Hispanic fourth-graders in public schools performed below NAEP Basic in the reading assessment. Black students are also overrepresented in the lowest level of the three below NAEP Basic subgroups. About 40 percent of the Black students performing below NAEP Basic fall into this lowest category, called below NAEP Basic Low. This means that 1 out of every 5 Black fourth-graders is in the lowest of the below NAEP Basic subgroups. For Hispanics, 1 out of every 6 students is in this lowest category.

In contrast, only 27 percent of all White fourth-graders in public schools performed below NAEP Basic in the reading assessment, and only 1 out of every 14 White fourth-graders (26 percent of White students performing below NAEP Basic) is in the lowest of the below NAEP Basic subgroups.

Results for other groups (including Asian, Native Hawaiian/Other Pacific Islander, American Indian/Alaska Native, and Two or more races) were too small to meet the NAEP reporting standard for robust estimation.

There were noticeable variations in performance among the three below NAEP Basic subgroups for every dimension of oral reading fluency and foundational skills. For example, fourth-graders in the below NAEP Basic Low subgroup read ORF passages at an average rate of 71 words correct per minute (WCPM), whereas those in the High subgroup read the same passages at an average rate of 108 WCPM.

In addition, there was more variation within the below NAEP Basic subgroups than within the three achievement levels (NAEP Basic, NAEP Proficient, and NAEP Advanced), especially for passage accuracy, and to a lesser degree for Word List WCPM and Passage WCPM. There was a 37 WCPM difference between the Low (71) and High (108) subgroups for passage reading, which is the same as the difference between NAEP Basic (123) and NAEP Advanced (160). See the details including the correlations between NAEP reading scores and oral reading variables here: https://nces.ed.gov/nationsreportcard/studies/orf/results.aspx.

Yes, they…

• Read a passage at about half the rate of a fourth-grader performing at the NAEP Proficient level (note that the NAEP Reading Assessment is timed);
• Misread 1 out of every 6 words—more likely content words critical to comprehension;
• Focused on individual words or phrases instead of the meanings of sentences and passages;
• Read in a monotone, indicating a lack of text comprehension; and
• Showed little knowledge of spelling-sound correspondence.

There are no NAEP achievement levels for oral reading fluency and foundational skills. Published norms for oral reading fluency Click to open pdf. compiled by the Behavioral Research and Teaching Program at the University of Oregon indicate that, among fourth-grade students in the winter of the school year, a student who reads 95 words correct per minute (WCPM) is at the 25th percentile, and a student who reads 71 WCPM is at the 10th percentile in oral reading fluency. Using these percentile scores as definitions for “poor” and “very poor” oral reading fluency, respectively, in the NAEP ORF study, students in the below NAEP Basic Medium subgroup showed poor fluency and students in the below NAEP Basic Low subgroup showed very poor fluency.

In the published norms for oral reading fluency Click to open pdf., fourth-graders’ average passage reading rate in 2017 was 120 WCPM. The average passage reading rate in the 2018 ORF study was also 120 WCPM.

No. They can read some words, but they do not read enough words with accuracy and appropriate speed to support their comprehension at the NAEP Basic level of the NAEP grade 4 reading assessment.

Fourth-graders performing at the NAEP Proficient and NAEP Advanced levels are still developing their fluency and word reading skills. As one indication of this, for passage reading, students performing at the NAEP Proficient level read an average of 142 words correct per minute (WCPM) and those at the NAEP Advanced level read an average of 160 WCPM. In comparison, based on the 2003 National Assessment of Adult Literacy (NAAL) ORF data Click to open pdf., a skilled adult reader reads orally at 166 to 178 WCPM.