Skip Navigation
Illustration/Logo View Quarterly by  This Issue  |  Volume and Issue  |  Topics
Education Statistics Quarterly
Vol 2, Issue 4, Topic: Methodology
NAEP Scoring of Twelfth-Grade Informative Writing
By: Sheida White and Alan Venneman
 
This article was originally published as a NAEPfact. The sample survey data are from the National Assessment of Educational Progress (NAEP) 1998 Writing Assessment.
 
 

Overview

The National Assessment of Educational Progress (NAEP) 1998 Writing Assessment measured student writing performance at the 4th, 8th, and 12th grades. Scoring guides for three different writing purposes for each grade allowed scorers to objectively evaluate students' work. This issue of NAEPfacts includes a 12th-grade persuasive writing scoring guide, along with samples of student work at each of six levels of performance on the scoring guide. The scorers of the NAEP 1998 Writing Assessment used a scoring method described as "focused holistic scoring." This approach combines holistic and "primary trait" scoring. A strict holistic approach to the scoring of writing treats a writing task as a "springboard" for writing. A particular writing task is given to students as a stimulus to engage them and inspire them to write, and students'


Introduction

The NAEP Writing Framework, developed by the National Assessment Governing Board (NAGB 1997, pp. 5-11), determined that the NAEP 1998 Writing Assessment should require students in the 4th, 8th, and 12th grades to write for three different purposes: narrative, informative, and persua-sive. Student performance would be evaluated on the basis of responses to a variety of different topics within each purpose.

Most 12th-graders received two writing topics, or "prompts," and were given 25 minutes to write on each while some received one 50-minute topic. Each topic was classified as either narrative, informative, or persuasive. Twenty-three topics were used in the 12th-grade assessment-twenty 25-minute topics and three 50-minute topics. Three 25-minute topics have been released to the public.1

Student writing samples were assessed according to a scoring guide that established six levels of student performance for each grade and writing purpose, ranging from "Unsatisfactory" to "Excellent."2 This issue of NAEPfacts includes the complete text of the 12th-grade persuasive scoring guide (figure 1), examples of persuasive writing by 12th-graders at each of the six levels, and a discussion of how the scoring guide applies to the six student writing samples.

back to top

Focused Holistic Scoring

The scorers of the NAEP 1998 Writing Assessment used a scoring method described as "focused holistic scoring." This approach combines holistic and "primary trait" scoring. A strict holistic approach to the scoring of writing treats a writing task as a "springboard" for writing. A particular writing task is given to students as a stimulus to engage them and inspire them to write, and students' responses are scored in terms of the overall writing quality. "Primary trait" writing scoring, on the other hand, is concerned with how well students respond to a specific topic. For example, if students are asked to write about whether they like adventure movies, students who do not address the topic of adventure movies will receive lower scores than those who do.

The "focused holistic scoring" approach used by NAEP, as with all holistic approaches, requires scorers to rate the overall quality of the writing, regardless of how students choose to respond to specific aspects of a given topic. In contrast to some holistic approaches to writing scoring that offer very general guidelines, NAEP scorers were given detailed scoring guides that focused their attention on specific characteristics of students' writing (organization, development, syntax, and mechanics). In this sense, the "traits" of writing now at issue for NAEP scoring of writing have shifted from topic-related traits of student responses to traits associated with overall quality of writing.

The basic assumptions of the NAEP focused holistic scoring approach are given below:

  • Each of the factors involved in writing is related to all others and no one factor can be separated from the others.
  • A writer is entitled to make some mistakes, given the 25-minute time limit, the lack of recourse to a dictionary, and the lack of time for reviewing and editing.
  • Scorers should read each response as a whole-without focusing on each mistake (but still being aware of them)-to judge the level of writing ability demonstrated by the student.
  • After thorough training on the scoring of responses written on a given task, scorers should quickly read an entire response and assign a score based on the total impression conveyed by the response.
  • Scorers should ignore their personal standards of what constitutes good writing and embrace the criteria of the scoring guide.
  • Scorers should read supportively rather than critically.
Figure 1.-Twelfth-grade persuasive writing scoring guide

Figure 1.-Twelfth-grade persuasive writing scoring guide

SOURCE: U.S. Department of Education, National Center for Education Statistics, (1999) The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462), p. 145.

back to top

Persuasive Writing

Persuasive writing focuses on the reader. Its primary aim is to influence others to take some action or bring about change. Persuasive writing may contain great amounts of information-facts, details, examples, comparisons, statistics, or anecdotes-but its main purpose is not simply to inform but to persuade. This type of writing involves a clear awareness of what arguments might most affect the audience being addressed. Writing persuasively also requires use of critical thinking skills such as analysis, inference, synthesis, and evaluation.

Persuasive writing is called for in a variety of situations. It may involve making a response to a request for advice by giving an opinion and providing sound reasons to support it. It may also involve presenting an argument in a way that a particular audience will find convincing. When there is opposition, persuasive writing may entail refuting arguments that are contrary to the writer's point of view.

In all persuasive writing, authors must choose the approach they will use. They may, for instance, use emotional or logical appeals or an accommodating or demanding tone. Regardless of the situation or approach, persuasive writers must be concerned with having a particular desired effect upon their readers, beyond merely adding to knowledge of the topic presented.

Twelfth-grade students were given a number of persuasive topics on which to write in the NAEP 1998 Writing Assessment. The same persuasive scoring guide was used for all of these topics. Because most students only had 25 minutes to create a writing sample, NCES did not strongly differentiate the narrative, informative, and persuasive scoring guides. For the most part, all three types of writing were scored according to the same criteria.

This issue of NAEPfacts presents 12th-grade writing samples at all six response levels, from "Unsatisfactory" to "Excellent," that students wrote to advocate a position on the efficacy of voting, either for or against. The complete text of the topic on which students were asked to write is given below:

     One Vote

     Your school is sponsoring a voter registration drive for 18-year-old high school students. You and three of your friends are talkingn about the project. Your friends say the following:

     Friend 1: "I'm working on the young voters' registration drive. Are you going to come to it and register? You'rea ll 18, so you can do it. We're trying to help increase the number of young people who vote and it shouldn't be too hard - I read that the percentage of 18- to 20-year-olds who vote increased in recent years. We want to keep that percentage up."

     Friend 2: "I'll be there. People should vote as soon as they turn 18. It's one of the responsibilities of living in a democracy."

     Friend 3:"I don't know if people should even bother to register. One vote in an election isn't going to change anything."

     Do you agree with friend 2 or 3? Write a response to your friends in which you explain whether you will or will not register to vote. Be sure to explain why and support your position with examples from your reading or experience. Try to convince the friend with whom you disagree that your position is the right one.

1. Sample "Unsatisfactory" response

Student response: I agree with #3 because if you want to vote go for it. Because it is you choice.

The "Unsatisfactory" rating was given to 4 percent of the responses to this prompt. As the scoring guide (figure 1) indicates, such responses were sometimes so unclear that the reader could not tell what position the student was taking. Other responses rated "Unsatisfactory" were extremely undeveloped. For example, in the response shown above, the student only states that he or she agrees with one of the three friends in the reported conversation and goes no further.

2. Sample "Insufficient" response

Student response: It is very Important that you would go out and vote. If everybody thought like that anyone could become president It is also important because who we pick will run or lead our nation for the next four years. We dont want just anyone up there, we want the best man to do the job. Or If voting for something else such as governor or senator, It dont matter. This is a priviledge given to us and we should take it not abuse it. People who would not or dont care to vote are just to lazy to go and vote.

The "Insufficient" rating was given to 21 percent of the responses to this prompt. Such responses were lacking either in organization or development (support of a position with reasons). In the "Insufficient" response shown above, the student does not justify his or her position beyond saying that it matters who gets elected.

3. Sample "Uneven" response

Student response: I would agree with Freind 2 because everyone should vote to support what they feel is nessecary. Also Freind 3 dosan't know what he or she is talking about because 1 vote can defenatly make a diffrence. I think I'm going to vote because if something were to happen like a new tax that I did not want my vote could have prevented that. Freind 3 can change alot just by his one vote so he should register, the reason for this is if 50 people voted on something and were all in favor for it and 49 were not in favor and he and I were with the 49 that were against it but did not register. If we would have registered it could have made it 51 people against and 50 for it.

The "Uneven" rating was given to 30 percent of the responses to this prompt. In such responses, students attempted to provide an argument supported with reasons, but faltered through lack of organization, problems with grammar that interfered with understanding, or incomplete development. In the response shown above, the student provides a somewhat undeveloped argument, despite the example at the end to illustrate how one vote can make a difference. The student jumps from the point that "everyone should vote to support what they feel is nessecary" to the statement that "1 vote can defenatly make a diffrence" without developing either point.

4. Sample "Sufficient" response

Student response: I think friend 2 is right. I believe that every single person's vote can help make a difference. It is important that we vote for who will lead our country, cities, counties and parishes. Our right to vote is our way of getting what we think our community deserves. The right to vote is your voice in the government. Many people who don't even vote complain about government leaders. But I say how can you complain if you didn't voice your opinion on who you think has the capability and skills to be a good leader. Your vote, along with others who didn't vote, could have made the difference. If no one voted our country would not have democracy. We could be lead by someone like Hitler or Mussolini. We as Americans have a choice. We should all take advantage of that right, to choose who will lead us. Who we choose to run our government has a direct effect on us. We should all be will to try to choose who's right and who's for the people.

The "Sufficient" rating was given to 32 percent of the responses to this prompt. In the "Sufficient" response shown above, the student organizes reasons into a complete, clear argument. Though the reasons are not developed with many details, the paper is organized and unified. The student connects points to build an argument: "Many people who don't even vote complain about government leaders. But I say how can you complain if you didn't voice your opinion on who you think has the capability and skills to be a good leader." The control of language is noticeably better than in responses that received ratings below "Sufficient." Some problems with mechanics, especially in the last sentence of the essay, do not impede the overall clarity and unity of the paper.

5. Sample "Skillful" response

Student response:

I would agree with friend 1 but in a slightly different way. Voting isn't a responsibility it's an opportunity. It is a way to show support for someone or something that you believe in. One of the great things about this country is that we have the right to vote and this right should not be taken for granted.

Friend 3 is somewhat right in the sense that one vote really won't make much of a difference (especially in a presidential election). However, if everyone used this as an excuse not to vote then the true beliefs of the general public would remain hidden from the government in which case they would do whatever they wanted because people wouldn't tell them what they think they should do.

It is becoming more important that young people vote because most of the registered voters are older. In order to get an accurate representation of what all citizens want then it is necessary for everyone to be an active voter.

The electoral college, in a way, discourages many people from voting because it eliminates the "one man, one vote" rule. It is very likely that many people will think that their vote makes little or no difference at all. Stories of electors that don't even vote for their pledged candidate do not help peoples' opinions on voting.

The "Skillful" rating was given to 10 percent of the responses to this prompt. In these responses, students elaborated reasons with details or examples in some, but not all, of the response and used transitions to connect ideas. In the "Skillful" response shown above, the student introduces the theme in the first paragraph: "Voting isn't a responsibility it's an opportunity." The student then points out why it is important to vote: to make the "beliefs of the general public" clear (second paragraph) and "to get an accurate representation of what all citizens want" (third paragraph).

6. Sample "Excellent" response 3

Student response: Whether a single person's vote makes a difference in an election is irrelevant. A democratic nation is one that recognizes an individual right to think and formulate an opinion, and voting is a manifestation of that right.

Mankind, the acknowledged ruler of the Earth, has little advantage over the other life-forms he shares existence with. As pointed out in the play Inherit the Wind, the horse is swifter, the mosquito more prolific, even a simple sponge is more durable. What separates mankind from other species is his simple brain-power: his ability to think.

The founding fathers of America recognized the fatal flaw of other nations - foolish monarchs who claimed absolute authority over their subjects. Dictatorial societies have the same root cause of their downfall - the attempts of squelching out personal opinion.

Voting celebrates the freedom the nation received on July 4, 1776 - voting is not a duty or a chore, it is a priveledge that we as humans have as our only advantage. We have the right and fortunautely because of our democratic society, the freedom to think.

The "Excellent" rating was given to 3 percent of the responses to this prompt. Students who wrote "Excellent" responses consistently elaborated reasons with details or examples, used transitions throughout, and often showed greater control over language (fewer errors and greater variety of sentence structure) than papers at the "Skillful" level. In the response shown above, the student provides a consistent, elaborated argument and demonstrates a command of rhetoric unusual even for an "Excellent" response to this prompt.

back to top

Conclusion

The scoring guides used in the NAEP 1998 Writing Assessment set six possible levels of writing performance for students, from "Unsatisfactory" to "Excellent." Among 12th-graders who wrote on the "One Vote" persuasive writing topic, 4 percent were rated "Unsatisfactory," 21 percent were rated "Insufficient," 30 percent were rated "Uneven," 32 percent were rated "Sufficient," 10 percent were rated "Skillful," and 3 percent were rated "Excellent."

Scoring guides, or "rubrics," are a widely used means of ensuring objective scoring for student work that requires a judgment of quality. Teachers working with scoring guides in the classroom can use the guides not only to evaluate student work but also to explain to students where their work needs improvement. 3

back to top


Footnotes

1 The three topics are available in The NAEP 1998 Writing Report Card for the Naional and the States (Greenwald et al. 1999). Additional information is available from the "Sample Questions" section of the NAEP Home Page: http://nces.ed.gov/nationsreportcard/ITMRLS/

2 The same basic guide was used for all three writing purposes, with some modifications.

3 For more information on the use of scoring guides or rubrics, see Moskal (2000). The ERIC Clearinghouse has a discussiion of rubrics, a bibliography, and additional links at http://ericae.net/faqz/rubrics/scoring_rubrics.htm

back to top

References

Greenwald, E.A., Persky, H.R., Campbell, J.R., and Mazzeo, J. (1999). The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462). U.S. Department of Education, National Center for Education Statistics. Washington, DC: U.S. Government Printing Office.

Moskal, B.M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research & Evaluation, 7(3). Available: http://ericae.net/pare/getvn.asp?v=7&n=3

National Assessment Governing Board. (1997). Writing Framework and Specifications for the 1998 National Assessment of Educational Progress. Washington, DC: Author. Available: http://www.nagb.org

back to top

For technical information, see

Greenwald, E.A., Persky, H.R., Campbell, J.R., and Mazzeo, J. (1999). The NAEP 1998 Writing Report Card for the Nation and the States (NCES 1999-462).

Author affiliations: S. White, NCES; A. Vanneman, Education Statistics Services Institute (ESSI).

For questions about content, contact Sheida White (sheida.white@ed.gov).

To obtain this NAEPfact (NCES 2000-488), call the toll-free ED Pubs number (877-433-7827) or visit the NCES Web Site (http://nces.ed.gov).

back to top