Interpreting NAEP Writing ResultsReporting the Assessment - Scale Scores and Achievement Levels Description of Writing Performance by Item Maps for Each Grade Exclusion Rates and Assessment Results Overview of the AssessmentAn ongoing assessment of student writing achievement administered to samples representative of the nation's students, the NAEP writing assessment reflects content specified by the National Assessment Governing Board (NAGB) framework, informed by expert perspectives about writing and its measurement. Read more about what the assessment measures, how it was developed, who took the assessment and how the assessment was administered. The 2002 writing results presented on the web site are for public- and nonpublic-school students in the nation at grades 4, 8, and 12 and for public-school students in participating states and jurisdictions at grades 4 and 8. In addition to the results from the 2002 writing assessment, results are presented from 1998 for the nation at all three grades and for participating states and other jurisdictions at grade 8. There was no state-level assessment at grade 4 in 1998. Although results were presented by region of the country (Northeast, South, Central, and West) in previous reports, regional data are not presented in this year's report because uneven school response rates in two of the regions made the comparative data less reliable than in the past. In some cases poor response was obtained from the samples of schools in states that were not participating in the NAEP state assessment program. Reporting the Assessment - Scale Scores and Achievement LevelsThe results of student performance on the NAEP writing assessment are presented on this web site in two ways: as scores on the NAEP writing scale and as the percentages of students attaining NAEP writing achievement levels. Scores on the NAEP writing composite scale range from 0 to 300; the three writing achievement levels are Basic, Proficient, and Advanced. The scale scores represent what students know and can do, while the achievement level results indicate the degree to which student performance meets expectations of what they should know and be able to do. The NAEP writing scales are composed of subscales, one for each of three fields of writing: narrative, informative, and persuasive. The composite scales were developed at each grade (4, 8, and 12) by weighting the separate subscales, based on the relative importance of each content area in the NAEP writing framework. While the scale score ranges are identical, the scale was derived independently at each grade. Also, scales were weighted differently at different grades in determining the overall scale. Therefore, average scale scores cannot be compared across grades. For example, equal scale scores on the grade 4 and grade 8 scales do not imply equal levels of writing achievement. Achievement level results are presented in terms of writing achievement levels as adopted by the National Assessment Governing Board (NAGB), and are intended to measure how well students' actual achievement matches the achievement expected of them. For each grade tested, NAGB has adopted three achievement levels: Basic, Proficient, and Advanced. Cut scores corresponding to the achievement levels are placed on the writing scales, resulting in four ranges: below Basic, Basic, Proficient, and Advanced. The Board established its achievement levels in 1996 based upon the writing content framework. The standard-setting process involving a broadly representative group of educators and interested citizens from across the nation who were asked to judge what students should know and be able to do relative to the content reflected in the NAEP writing framework. As provided by law, NCES has determined that the achievement levels are to be used on a trial basis and should be interpreted and used with caution. However, both NCES and the NAGB Board believe these performance standards are useful for understanding trends in student achievement. Description of Writing Performance by Item Maps for Each GradeThe performance of fourth-, eighth-, and twelfth-graders can be illustrated by maps that position question descriptions along the NAEP writing scale. The descriptions used on these maps focus on the writing skill or knowledge needed to answer the question. For multiple-choice questions, the description indicates the skill or knowledge demonstrated by selection of the correct option; for constructed-response questions, the description takes into account the skill or knowledge specified by the different levels of scoring criteria for that question. Approximately 25 writing questions per grade have been selected and placed on an item map for grade 4, grade 8, and grade 12. NAEP Reporting GroupsResults are provided for groups of students defined by shared characteristics—gender, race or ethnicity, school's type of location, Title I participation, eligibility for free/reduced-price school lunch, and type of school. Based on participation rate criteria, results are reported for subpopulations only when sufficient numbers of students and adequate school representation are present. The minimum requirement is at least 62 students in a particular subgroup from at least five primary sampling units (PSUs). However, the data for all students, regardless of whether their subgroup was reported separately, were included in computing overall results. Definitions of the subpopulations are presented below. GenderResults are reported separately for males and females. Race/EthnicityIn all NAEP assessments, data about student race/ethnicity is collected from two sources: school records and student self-reports. Previously, NAEP has used student self-reported race as the primary race/ethnicity reporting variable. In 2002, it was decided to change the student race/ethnicity variable highlighted in NAEP reports. Starting in 2002, school-recorded race will become the race/ethnicity variable presented in NAEP reports. Information based on student self-reported race/ethnicity will continue to be reported in the NAEP Data Tool. In order to allow comparisons across years, both the 1998 and 2002 writing assessment results presented in this report are based on school-reported information for six mutually exclusive racial/ethnic categories: White, Black, Hispanic, Asian/Pacific Islander, American Indian (including Alaska Native), and Other. Students who identified with more than one of the first five categories or had a background other than the ones listed were categorized as Other. Type of LocationResults from the 2002 assessment are reported for students attending schools in three mutually exclusive location types: Central city: A central city is a city of 50,000 or more that is the largest in its metropolitan area, or can otherwise be regarded as "central." The term means "a city that is central," not "the central part of a city" or the "inner city." Note that central cities encompass wider areas than what is commonly referred to as "the inner city." Urban fringe/large town: The urban fringe category includes any incorporated place, census designated place, or non-place territory within a CMSA or MSA of a large or mid-sized city that is defined as urban by the U.S. Census Bureau, but that does not qualify as central city. A large town is defined as a place outside a CMSA or MSA with a population greater than or equal to 25,000. Rural/small town: Rural includes all places and areas with populations of less than 2,500 that are classified as rural by the U.S. Census Bureau. A small town is defined as a place outside a CMSA or MSA with a population of less than 25,000, but greater than or equal to 2,500. New methods have been put in place by NCES to identify the type of location assigned to each school in the Common Core of Data (CCD). The new methods are used in order to improve the quality of the assignments, and they take into account more information about the exact physical location of the school. The variable was revised in NAEP beginning with the 2000 assessments. Therefore results for each type of location are not compared across years. Title I ParticipationBased on available school records, students were classified either as currently participating in a Title I program, receiving Title I services, or as not receiving such services. The classification applies only to the school year when the assessment was administered (i.e., the 2001—02 school year) and is not based on participation in previous years. If the school does not offer any Title I programs or services, all students in that school would be classified as not participating. Eligibility for Free/Reduced-Price School LunchAs part of the Department of Agriculture's National School Lunch Program, schools can receive cash subsidies and donated commodities in turn for offering free or reduced-price lunches to eligible children. Based on available school records, students were classified as either currently eligible for the free/reduced-price school lunch or not eligible. Eligibility for free and reduced-price lunches is determined by students' family income in relation to the federally established poverty level. Students whose family income is at or below 130 percent of the poverty level qualify to receive free lunch, and students whose family income is between 130 percent and 185 percent of the poverty level qualify to receive reduced-price lunch. The classification applies only to the school year when the assessment was administered (i.e., the 2001—02 school year) and is not based on eligibility in previous years. If school records were not available, the student was classified as "Information not available." If the school did not participate in the program, all students in that school were classified as "Information not available." Parents' Highest Level of EducationEighth-grade students who participated in the NAEP 2002 writing assessment, including those in the state assessments and the Trial Urban District Assessment, were asked to indicate the highest level of education completed by each parent. The question was not posed to fourth-graders. Five response options were offered: did not finish high school, graduated from high school, some education after high school, graduated from college, or "I don't know." The highest level of education reported for either parent was used in the analysis of this response. Type of SchoolResults are reported by the type of school that the student attends—public or nonpublic. Nonpublic schools include Catholic and other private schools. Because they are funded by federal authorities, not state/local governments, Bureau of Indian Affairs (BIA) schools and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) are not included in either the public or nonpublic categories; they are included in the overall national results. Results are EstimatesThe average scores and percentages presented on this web site are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented on this web site are available through the NAEP Data Tool. Statistical SignificanceThe differences between scale scores and between percentages discussed in the results on this web site take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude or the difference between the group average scores or percentages and the standard errors of those statistics. On the NAEP web site, differences between scores or between percentages are noted as such only when they are significant from a statistical perspective. All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude of the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policy makers, educators, and the public. Exclusion Rates and Assessment ResultsAlthough 50 jurisdictions participated in the 2002 writing assessment at grade 4, and 50 participated at grade 8, not all met minimum school participation guidelines for reporting their results. (See appendix A of the Writing Report Card for details on participation and reporting guidelines.) Results from the 2002 assessment are not included for Illinois and Wisconsin at grades 4 and 8, and for Minnesota at grade 8, because they did not meet the minimum weighted school participation rate of 70 percent. Jurisdictions that did not meet one or more of the other participation guidelines are noted in each of the tables on the web site. To ensure that the samples in each state are representative, NAEP has established policies and procedures to maximize the inclusion of all students in the assessment. Every effort is made to ensure that all selected students who are capable of participating meaningfully in the assessment are assessed. While some students with disabilities and/or limited English proficient (SD and/or LEP) students can be assessed without any special procedures, others require accommodations to participate in NAEP. Still other SD and/or LEP students selected by NAEP may not be able to participate. Local school authorities determine whether SD/LEP students require accommodations or shall be excluded because they cannot be assessed. The percentage of SD and/or LEP students who are excluded from NAEP assessments varies from one jurisdiction to another and within a jurisdiction over time. If excluded students are less proficient writers, variations in exclusion rates could have an impact on average writing scores or score gains within jurisdictions. NCES is currently sponsoring ongoing research on the potential impact of changes in exclusion rates on changes in average writing performance. The preliminary findings from the research suggest that the potential impact on writing scores is minimal. Further discussion of this research is presented in appendix A of the 2002 Writing Report Card. Cautions in InterpretationsUsers of this web site are cautioned against interpreting NAEP results in a causal sense. Inferences related to subgroup performance or to the effectiveness of public and nonpublic schools, for example, should take into consideration the many socioeconomic and educational factors that may also impact performance. The NAEP writing scale makes it possible to examine relationships between students' performance and various background factors measured by NAEP. However, a relationship that exists between achievement and another variable does not reveal its underlying cause, which may be influenced by a number of other variables. Similarly, the assessments do not reflect the influence of unmeasured variables. The results are most useful when they are considered in combination with other knowledge about the student population and the educational system, such as trends in instruction, changes in the school-age population, and societal demands and expectations. Return to main results.
|
