Skip Navigation
Illustration/Logo View Quarterly by  This Issue  |  Volume and Issue  |  Topics
Education Statistics Quarterly
Vol 2, Issue 4, Topic: Methodology
NAEP Scoring of Eighth-Grade Informative Writing
By: Sheida White and Alan Venneman
 
This article was originally published as a NAEPfact. The sample survey data are from the National Assessment of Educational Progress (NAEP) 1998 Writing Assessment.
 
 

Overview

The National Assessment of Educational Progress (NAEP) 1998 Writing Assessment measured student writing performance at the 4th, 8th, and 12th grades. Scoring guides for each grade allowed scorers to objectively evaluate students' work. This issue of NAEPfacts includes a 8th-grade narrative scoring guide, along with samples of student work at each of six levels of performance on the scoring guide.


Introduction

The NAEP Writing Framework, developed by the National Assessment Governing Board (NAGB 1997, pp. 5-11), determined that the NAEP 1998 Writing Assessment should require students in the 4th, 8th, and 12th grades to write for three different purposes: narrative, informative , and persuasive. Student performance would be evaluated on the basis of responses to a variety of different topics within each purpose.

Most students in the eighth grade received two writing topics, or "prompts," and were given 25 minutes to write on each, although some received a single 50-minute topic. Each topic was classified as either narrative, informative, or persuasive. Twenty-three different topics were used in the 8th-grade assessment: twenty 25-minute topics and three 50-minute topics. Three 25-minute topics have been released to the public.1

Student writing samples were assessed according to a scoring guide that established six levels of performance, ranging from "Unsatisfactory" to "Excellent."2 This issue of NAEPfacts includes the complete text of the eighth-grade "informative" scoring guide (figure 1), examples of informative writing by eighth-graders at each of the six levels, and a discussion of how the scoring guide applies to the six student writing samples.

Figure 1.-Eighth-grade persuasive writing scoring guide

Figure 1.-Eighth-grade persuasive writing scoring guide

SOURCE: U.S. Department of Education, National Center for Education Statistics, (1999) The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462), p. 145.

back to top

Focused Holistic Scoring

The scorers of the NAEP 1998 Writing Assessment used a scoring method described as "focused holistic scoring." This approach combines holistic and "primary trait" scoring. A strict holistic approach to the scoring of writing treats a writing task as a "springboard" for writing. A particular writing task is given to students as a stimulus to engage them and inspire them to write, and students' responses are scored in terms of the overall writing quality. The "primary trait" method of scoring writing, on the other hand, is concerned with how well students respond to a specific topic. For example, if students are asked to write about whether they like adventure movies, students who do not address the topic of adventure movies will receive lower scores than those who do.

The "focused holistic scoring" approach used by NAEP, as with all holistic approaches, requires scorers to rate the overall quality of the writing, regardless of how students choose to respond to specific aspects of a given topic. In contrast to some holistic approaches to writing scoring that offer very general guidelines, NAEP scorers were given detailed scoring guides that focused their attention on specific characteristics of students' writing (organization, development, syntax, and mechanics). In this sense, the "traits" of writing now at issue for NAEP scoring of writing have shifted from topic-related traits of student responses to traits associated with overall quality of writing.

The basic assumptions of the NAEP focused holistic scoring approach are given below:

  • Each of the factors involved in writing is related to all others and no one factor can be separated from the others.
  • A writer is entitled to make some mistakes, given the 25-minute time limit, the lack of recourse to a dictionary, and the lack of time for reviewing and editing.
  • Scorers should read each response as a whole-without focusing on each mistake (but still being aware of them)-to judge the level of writing ability demonstrated by the student.
  • After thorough training on the scoring of responses written on a given task, scorers should quickly read an entire response and assign a score based on the total impression conveyed by the response.
  • Scorers should ignore their personal standards of what constitutes good writing and embrace the criteria of the scoring guide.
  • Scorers should read supportively rather than critically.
back to top


Informative Writing

Informative writing focuses primarily on the subject-matter element in communication. This type of writing is used to share knowledge and to convey messages, instructions, and ideas. Like all writing, informative writing may be filtered through the writer's impressions, understanding, and feelings. Used as a means of exploration, informative writing helps both the writer and the reader to learn new ideas and to reexamine old conclusions.

Informative writing may also involve reporting on events or experiences, or analyzing concepts and relationships, including developing hypotheses and generalizations. Any of these types of informative writing can be based on the writer's personal knowledge and experience or on information newly presented to the writer that must be understood in order to complete a task. Usually, informative writing involves a mix of the familiar and the new, and both are clarified in the process of writing. Depending on the task, writing based on either personal experience or secondary information may span the range of thinking skills from recall to analysis to evaluation.

Eighth-grade students were given a number of informative topics in the NAEP 1998 Writing Assessment. The same informative scoring guide was used for all of these topics. This issue of NAEPfacts presents eighth-grade student writing samples at all six response levels, from "Unsatisfactory" to "Excellent," that students wrote about a proposed television show for teenagers. The complete text of the topic is as follows:

     Designing a TV Show

     A public television network is seeking ideas for a new series of shows that would be educational for teenagers. The series will include ten one-hour episodes and will be shown once a week. Some of the titles under consideration are:

     "Great Cities of the World"

     "Women in History"

     "Nature Walks"

     "American Legends"

     Choose one of these titles. Write a letter to the network president describing your ideas for a new educational series. In your letter, describe what one episode might be like. Use specific examples of what information you would include in the episode so that network president will be able to imagine what the series would be like.

1. Sample "Unsatisfactory" response

Student response:

Dear President,

I would like to do a brochure, on "Great Cities of the World" I need your opinion should I do it on New York, Tokyo, Tiawan, Los Angelos, or should I do all of them?

Always
Student

The "Unsatisfactory" rating was given to 3 percent of the responses to this topic. As the scoring guide (figure 1) indicates, such responses were very undeveloped or very poorly written. In the "Unsatisfactory" response shown above, the student chooses one of the series titles provided in the topic and asks what to include, without presenting his or her ideas about what to show on the television series.

2. Sample "Insufficient" response

Student response:

Dear President

I think you should do the series on "Great City's of the World." If you did the series off of that title it would be best. You would get to learn about all the cities instead of just one city. Because teenagers could learn about other cities in other countries. That's why I think you should do the series on "Great City's of the World."

The "Insufficient" rating was given to 13 percent of the responses to this topic. Such responses supplied only minimal information about the student's choice of an educational television series. In the example shown above, the student provides a justification for the series: "You would get to learn about all the cities instead of just one city." However, the student does not develop that justification by describing the substance of the show.

3. Sample "Uneven" response

Student response:

Dear Network President,

I think you should do a show on American legends. You can tell about real people like George Washington or Abraham Lincon. You might want to consider using fictional characters such as Paul Bunyan or Johnny Appleseed. You might want to do shorter section on all of the less popular Presidents like Teddy Roosevelt or Woodrow Wilson.

I would put in how George Washington helped win the Revolutionary War or how he made a good President You could also tell how John F. Kennedy was assasianated or how Abraham Lincon helped in the Cival War.

The "Uneven" rating was given to 40 percent of the responses to this topic. In many of these responses, students mentioned a few specific elements to be presented on the television series, but listed rather than developed them. In the "Uneven" paper shown above, the student enumerates various "American Legends" to be presented, along with an identifying detail or two about George Washington, John F. Kennedy, and Abraham Lincoln, for example: "You could also tell how John F. Kennedy was assasianated or how Abraham Lincon helped in the Cival War." The student, however, does not develop points, and his or her command of the mechanics of writing is uneven.

4. Sample "Sufficient" response

Student response:

Dear Mr. President,

I think you should have a show about "Women in history." A lot of people want to know about women and what they've done to help our country. There have been many women heroes, and they should be recognized. You could do the show like Wishbone, except all the shows be about women in history instead of characters from a book. An idea for a show is Anne Frank. You could go to the place where they hid for so long and do the show right there. Everyone will get the chance to see how Anne lived. A lot of people haven't heard or seen her story. Well, it's time they do! So, please take into consideration my ideas and respond when you make your decision.

Students at the upper score levels ("Sufficient" or better) provided organized responses with illustrative details. Some students provided descriptions of an entire episode, down to the dialogue and camera angles.

The "Sufficient" rating was given to 34 percent of the responses to this topic. Such responses were organized and provided some details. In the response shown above, the student's writing is clear, accurate, and organized.

5. Sample "Skillful" response

Student response:

Dear Network President,

I think that I have a new show for your network. It's called Great Cities of the World. The show is about four teenagers, around the ages of fourteen to seventeen who travel around the world. In each show they travel to two cities. When they arrive in the city they will first talk about the cities history and what it is like now in the present. They talk about some of the tradions of the city. For example if the students went to Paris, France they would talk about France's past and some of the things they do in there daily lives. They could talk about the people, what they look like and their styles. To keep the show interesting you can show things such as we learn how to say a word from their language or meet many different people from their city. Also to keep the show interesting they can have problems

The "Skillful" rating was given to 8 percent of the responses to this topic. In such responses, students used detail and elaboration in parts of the response, with transitions to connect ideas. In the response shown above, the student specifies who will be the narrators of the show and the order in which information will be presented: "The show is about four teenagers, around the ages of fourteen to seventeen who travel around the world. In each show they travel to two cities. When they arrive in the city they will first talk about the cities history and what it is like now in the present." The student also uses the example of Paris as the subject for one show. The student uses complex sentences and transitions (such as "When they arrive in the city. . . . For example. . . .") to tie points together and lead the reader through the essay.

6. Sample "Excellent" response3 3

Student response: Dear Network president,

Hello! I am a young teenager and I think that teenagers these days would like to see something educational. I think a good idea for a t.v. show would be "Great Cities of the World.

¶For example, one episode could be about Chicago and tell famous places you could visit. One place could be the Sears Tower in which a camera could show people going up in an elevator and then seeing the view of downtown Chicago. ¶Another place the t.v. show could go to is the Shed Aquarium. In it are many types of ocean life that interesting to see up close. They could also go to the art museum and look at famous paintings. ¶Just for fun, the show could go to F.A.O. Schwartz, a large toy store with many toys you can play with. As a matter of fact, you could just go shopping period. Chicago is known for its many stores ¶Then you could take a trip to a restaurant such as Ed Debivic's or Planet Hollywood, just to spice up the show a bit. ¶Now that I've explained where to go in Chicago, I'll tell you a little more about the set-up of the show. I think that you should have a host who is young, around fiftheen, energetic, and a spunky personality. She or he could act as the tour guide and show the viewers around each city. She could also explain the city's trademark's, such as the Sears Tower. I think that if you use a young person, it would attract young viewers.

¶ And last of all, I think the camera should look at the city as if it was viewer's eyes. For example, when you look around, you see things as you would see them, as if you were really there in Chicago, sight-seeing.

The "Excellent" rating was given to 2 percent of the responses to this topic, in which students used detail and development across the response. The "Excellent" response shown above describes an entire episode of a television series in detail. The student includes such details as how the camera would move: "One place could be the Sears Tower in which a camera could show people going up in an elevator and then seeing the view of downtown Chicago." He or she describes a wide variety of sights in Chicago with suggestions for how to present them. Points such as "I think the camera should look at the city as if it was viewer's eyes" enable the reader to visualize the show. This student shows good control of language; occasional minor errors do not interfere with meaning.

back to top

Conclusion

The scoring guides used in the NAEP 1998 Writing Assessment set six possible levels of writing performance for students, from "Unsatisfactory" to "Excellent." Among eighth-graders who wrote on the "TV Show" informative writing topic, 3 percent were rated "Unsatisfactory," 13 percent were rated "Insufficient," 40 percent were rated "Uneven," 34 percent were rated "Sufficient," 8 percent were rated "Skillful," and 2 percent were rated "Excellent."

Scoring guides, or "rubrics," are a widely used means of ensuring objective scoring for student work that requires a judgment of quality. Teachers working with scoring guides in the classroom can use the guides not only to evaluate student work but also to explain to students where their work needs improvement.4 4

back to top

Footnotes

1 The three topics are available in The NAEP 1998 Writing Report Card for the Naional and the States (Greenwald et al. 1999). Additional information is available from the "Sample Questions" section of the NAEP Home Page: http://nces.ed.gov/nationsreportcard/ITMRLS/

2 The same basic guide was used for all three writing purposes, with some modifications.

3The "" symbols in the sample are paragraph signs and reflect symbols placed in the text by the student.

4 For more information on the use of scoring guides or rubrics, see Moskal (2000). The ERIC Clearinghouse has a discussiion of rubrics, a bibliography, and additional links at http://ericae.net/faqz/rubrics/scoring_rubrics.htm

back to top

References

Greenwald, E.A., Persky, H.R., Campbell, J.R., and Mazzeo, J. (1999). The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462). U.S. Department of Education, National Center for Education Statistics. Washington, DC: U.S. Government Printing Office.

Moskal, B.M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research & Evaluation, 7(3). Available: http://ericae.net/pare/getvn.asp?v=7&n=3

National Assessment Governing Board. (1997). Writing Framework and Specifications for the 1998 National Assessment of Educational Progress. Washington, DC: Author. Available: http://www.nagb.org

For technical information, see

Greenwald, E.A., Persky, H.R., Campbell, J.R., and Mazzeo, J. (1999). The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462).

Author affiliations: S. White, NCES; A. Vanneman, Education Statistics Services Institute (ESSI).

For questions about content, contact Sheida White (sheida.white@ed.gov).

To obtain this NAEPfact (NCES 2000-506), call the toll-free ED Pubs number (877-433-7827) or visit the NCES Web Site (http://nces.ed.gov).

back to top