• Introduction
• Achievement Levels Provide Standards for Student Performance
• NAEP 2003 Mathematics Assessment Design
• Achievement-Level Results for Urban Districts
• Percentile Results for 2003
• How Various Groups of Students Performed in Mathematics
• Testing Status of Special-Needs Students Selected in NAEP Samples

Since 1969, the National Assessment of Educational Progress (NAEP) has been an ongoing nationally representative indicator of what American students know and can do in major academic subjects. Over the years, NAEP has measured students' achievement in many subjects, including reading, mathematics, science, writing, U.S. history, geography, civics, and the arts. In 2003, NAEP conducted national and state assessments in reading and mathematics at grades 4 and 8. NAEP is a project of the National Center for Education Statistics (NCES) within the Institute of Education Sciences (IES) of the U.S. Department of Education, and is overseen by the National Assessment Governing Board (NAGB).

In 2001, after discussion among NCES, NAGB, and the Council of the Great City Schools, Congress appropriated funds for a district-level assessment on a trial basis, similar to the trial for state assessments that began in 1990, and NAGB passed a resolution approving the selection of urban districts for participation in the Trial Urban District Assessment (TUDA), a special project within NAEP.

Representatives of the Council of the Great City Schools worked with the staff of NAGB to identify districts for the trial assessment. Districts were selected that permitted testing of the feasibility of conducting NAEP over a range of characteristics, such as district size, minority concentrations, federal program participation, socioeconomic conditions, and percentages of students with disabilities (SD) and limited-English-proficient (LEP) students.

By undertaking the TUDA, NAEP continues a tradition of extending its service to education, while preserving the rigorous sampling, scoring, and reporting procedures that have characterized prior NAEP assessments at both the national and state levels.

In 2002, five urban school districts participated in NAEP's first TUDA in reading and writing. In 2003, nine urban districts (including the original five) participated in the TUDA in reading and mathematics at grades 4 and 8: Atlanta City, Boston School District, Charlotte-Mecklenburg Schools, City of Chicago School District 299, Cleveland Municipal School District, Houston Independent School District, Los Angeles Unified, New York City Public Schools, and San Diego City Unified. Only public school students were sampled in the TUDA. Results for the District of Columbia public schools, which normally participate in NAEP's state assessments, are also reported (figure A).

Average mathematics scores are reported on a 0–500 scale. Figure A shows the average scores at both grades for the districts that participated in 2003. The average scores for public school students in the nation and for public school students attending schools located in large central cities are also shown for comparison. "Urban districts" refers to the 10 districts reported in this trial study. Eight of the 10 urban districts consist entirely of schools in cities with a population of 250,000 or more (i.e., large central cities as defined by NCES); two of them (Charlotte and Los Angeles) consist primarily of schools in large central cities, but also have from one-quarter to one-third of their fourth- and eighth-grade students enrolled in surrounding urban fringe or rural areas. All of the data for both districts were used to compare with data from large central cities and the nation.

At grade 4, the average score in Charlotte was higher than the average scores for the nation, large central cities, and the other participating districts. All participating districts at grade 4 except Charlotte had lower average scores than the average score for the nation. Compared with the average score in large central cities, the average scores in three districts (Houston, New York City, and San Diego) were not found to be significantly different, and the average scores in the remaining six districts were lower.

At grade 8, the average score in Charlotte was again higher than the average scores for the nation, large central cities, and the other participating districts, while the average scores for all other districts were lower than that for the nation. Students in New York City also scored higher, on average, than students in large central city public schools, while the average scores for students in Boston, Houston, and San Diego were not found to be significantly different from that in large central cities. The average scores in the remaining five districts were lower than the average score in large central cities.

Figure A. Average NAEP mathematics scores, grade 4 and grade 8 public schools: By urban district, 2003

Achievement levels are performance standards set by NAGB to provide a context for interpreting student performance on NAEP. These performance standards, based on recommendations from broadly representative panels of educators and members of the public, are used to report what students should know and be able to do at the Basic, Proficient, and Advanced levels of performance in each subject area and at each grade assessed.¹

The minimum scale scores for achievement levels are as follows:

As provided by law, NCES, upon review of a congressionally mandated evaluation of NAEP, has determined that achievement levels are to be used on a trial basis and should be interpreted and used with caution. However, both NCES and NAGB believe that these performance standards are useful for understanding trends in student achievement. NAEP achievement levels have been widely used by national and state officials.

The NAEP mathematics framework, which defines the content for the 2003 assessment, was developed through a comprehensive national consultative process and approved by NAGB. The mathematics framework calls for the assessment to include questions based on five mathematics content areas: (1) number sense, properties, and operations; (2) measurement; (3) geometry and spatial sense; (4) data analysis, statistics, and probability; and (5) algebra and functions.

In addition, the framework specifies that each question should measure one of three mathematical abilities. The three mathematical abilities specified by the framework are (1) conceptual understanding, (2) procedural knowledge, and (3) problem solving. The complete framework is available on the NAGB web site (http://www.nagb.org/pubs/pubs.html).

Results from the 2003 TUDA are reported for the participating districts for public school students at grades 4 and 8. The TUDA employed larger-than-usual samples within the districts, making reliable district-level data possible. The samples were also large enough to provide reliable estimates on subgroups within the districts, such as female students or Hispanic students.

It is NAEP's intent to assess all selected students from the target population. Beginning in 2002, SD and LEP students who require accommodations have been permitted to use them in NAEP, unless a particular accommodation would alter the skills and knowledge being tested. For example, students may not use calculators for questions not intended for calculator use. Because the representativeness of samples is ultimately a validity issue, NCES has commissioned studies of the impact of assessment accommodations on overall scores. One paper that explores the impact of two possible scenarios on NAEP is available on the web site (http://nces.ed.gov/nationsreportcard/pdf/main2002/statmeth.pdf).

At grade 4, the percentages of students in Charlotte performing at or above Basic, at or above Proficient, and at Advanced were higher than the corresponding percentages in both large central cities and the nation. The percentages of fourth-graders at or above Basic in Houston and New York City were higher than the percentage in large central cities.

At grade 8, the percentages of students in Charlotte at or above Proficient and at Advanced were higher than the corresponding percentages in both large central cities and the nation. The percentage of eighth-graders at or above Basic in Boston, Houston, New York City, and San Diego was not found to be different from the percentage in large central cities.²

Examining the performance of students at different locations (high, middle, and low) on the full student score distribution gives a more complete picture than examining the average score alone. The percentile indicates the percentage of students whose scores fell below a particular score. For example, to score above the 25th percentile nationally, a fourth-grade public school student would have had to score at least 215, compared to a fourth-grade public school student in a large central city who would have had to score at least 204.

At both grades 4 and 8, the scores for all of the districts except Charlotte were lower than those of public schools in the nation at the 25th, 50th, and 75th percentiles. At grade 4, the score at the 75th percentile for students in large central cities was lower than the score for Charlotte; not found to differ significantly from the scores for Houston, New York City, and San Diego; and higher than the scores in the remaining districts.

At grade 8, the score at the 75th percentile for students in large central cities was lower than that for Charlotte; not found to differ significantly from the scores for Boston, New York City, and San Diego; and higher than the scores in the remaining districts.

In addition to reporting the overall performance of assessed students, NAEP also reports on the performance of various subgroups of students. The performance of subgroups of students on the 2003 TUDA in mathematics can be compared with that of their counterparts in large central city public schools and the nation. In addition, this assessment serves as a baseline for future comparisons of students' performance in mathematics.

When reading these subgroup results, it is important to keep in mind that there is no simple, cause-and-effect relationship between membership in a subgroup and achievement in NAEP. A complex mix of educational and socioeconomic factors may interact to affect student performance.

At grade 8, the average scores for both male and female students in Charlotte were higher than the corresponding average score for large central cities. The average scores for both male and female eighth-graders in Boston, Houston, New York City, and San Diego were not found to differ significantly from the corresponding average scores in large central cities. Both male and female eighth-graders in Atlanta, Chicago, Cleveland, the District of Columbia, and Los Angeles had lower average scores than their counterparts in large central cities and in the nation.

Average score gaps between male and female students in mathematics. In 2003, male public school students in the nation scored higher, on average, than female students by 3 points at grade 4 and by 2 points at grade 8. At grade 4, the score gap between male and female students in the District of Columbia was the reverse of the gap in the nation and large central cities (i.e., female students outscored males). The score gap between male and female students for Los Angeles was wider than that in the nation. At grade 8, there was also a reversal of the score difference for male and female students in Atlanta, Boston, and the District of Columbia (i.e., female students outscored male students).

Achievement-level results by gender. The percentages of male and female students performing below Basic, at or above Basic, at or above Proficient, and at Advanced are presented below. At grade 4, the percentages of male and female students performing at or above Proficient in public schools nationally were higher than the percentages for all districts except Charlotte, where the percentages at or above Proficient were higher than those for the nation. When compared with male and female students in large central city public schools, higher percentages of both male and female fourth-grade students in Charlotte performed at or above Proficient. The percentages of fourth-grade male and female students performing at or above Proficient in Houston, New York City, and San Diego were not found to differ significantly from the corresponding percentages at or above Proficient in large central cities.

At grade 8, greater percentages of male students in Charlotte performed at or above Proficient than in public schools nationally and in large central cities. Greater percentages of female eighth-grade students in Charlotte and New York City performed at or above Proficient than those in large central city public schools. The percentages of eighth-grade male and female students in Boston and San Diego and eighth-grade male students in New York City were not found to differ significantly from the percentage at or above Proficient in large central cities. Lower percentages of male and female students in the other TUDA districts performed at or above Proficient than the percentages of their counterparts in large central city public schools.

At grade 4, the average scale scores for White students in Charlotte, the District of Columbia, and Houston; Black students in Boston, Charlotte, Houston, and New York City; and Hispanic students in Charlotte and Houston were higher than the corresponding scores in large central cities (table A). The average scores for fourth-grade White students in Boston, Chicago, and Cleveland; Black students in Chicago and the District of Columbia; and Hispanic students in Boston, the District of Columbia, Los Angeles, and San Diego were lower than the corresponding scores in large central cities.

At grade 8, the average scale scores for White students in Atlanta, Charlotte, and Houston; Black students in Charlotte, Houston, and New York City; and Hispanic students in Houston were higher than the corresponding scores in large central cities (table A). The average scores for eighth-grade White students in Cleveland; Black students in Atlanta, the District of Columbia, and Los Angeles; and Hispanic students in the District of Columbia, Los Angeles, and San Diego were lower than the corresponding scores in large central cities.

Average mathematics score gaps between selected racial/ethnic subgroups. At grade 4, the gaps between White students and Black students in Boston and New York City were narrower than that in large central cities; the gaps in Atlanta and the District of Columbia were wider than the gap between White students and Black students in large central cities. The gap between White students and Hispanic students was wider in the District of Columbia than the gap in large central cities.

At grade 8, the gap between White students and Black students in Cleveland was narrower than the gap in large central cities, and the gaps in Atlanta and Charlotte were wider than the gap between White students and Black students in large central cities. The gaps between White students and Hispanic students for eighth-graders were wider in Boston and San Diego than in large central cities and wider in Charlotte than in the nation. In Chicago, the gap between White students and Hispanic students was narrower than that in large central cities and the nation.

Achievement-level results by race/ethnicity. At grade 4, the percentages of students at or above the Proficient level were higher for White students in Atlanta, Charlotte, the District of Columbia, and Houston; Black students in Charlotte and New York City; and Hispanic students in Charlotte than the corresponding percentage in large central cities. The percentages of fourth-grade students at or above Proficient for White students in Boston, Chicago, and Cleveland; Black students in Chicago, Cleveland, and the District of Columbia; and Hispanic students in Boston, the District of Columbia, Los Angeles, and San Diego were lower than the corresponding percentage in large central cities.

At grade 8, the percentages of students at or above the Proficient level were higher for White students in Atlanta, Boston, Charlotte, and Houston and for Black students in Charlotte and New York City than that of their counterparts in large central cities. The percentages of eighth-grade students at or above the Proficient level for White students in Cleveland; Black students in Atlanta, the District of Columbia, and Los Angeles; and Hispanic students in Boston, the District of Columbia, Los Angeles, and San Diego were lower than the corresponding percentage in large central cities.

Mathematics performance by students' eligibility for free/reduced-price lunch. NAEP collects data on students' eligibility for free/reduced-price lunch as an indicator of economic status. In 2003, approximately 7 percent of fourth-graders and 6 percent of eighth-graders nationally attended schools that did not participate in the National School Lunch Program. Note that Cleveland chose to define all of its students as eligible for free or reduced-price lunch. Information regarding students' eligibility in 2003 was not available for 4 percent of fourth-graders and 6 percent of eighth-graders nationally. For information on the National School Lunch Program, see http://www.fns.usda.gov/cnd/lunch/default.htm.

At grade 4, the average scores for students eligible for free/reduced-price lunch in Charlotte, Houston, and New York City were higher than the average score for large central cities nationally. The average scores for eligible fourth-graders in Boston, Cleveland, and San Diego were not found to differ significantly from the average score for large central cities; the average scores for eligible students in Atlanta, Chicago, the District of Columbia, and Los Angeles were lower than the average score for eligible students in large central cities.

At grade 8, the average scores for students who were eligible for free/reduced-price lunch in Boston, Houston, and New York City were higher than the average score for large central cities. In Charlotte, Chicago, Cleveland, and San Diego, the average scores for eligible eighth-graders were not found to differ from that in large central cities. The average scores for eligible students in Atlanta, the District of Columbia, and Los Angeles were lower than the average score in large central cities.

Average mathematics score gaps between students who were eligible and those who were not eligible for free/reduced-price lunch. In 2003, public school students in the nation who were not eligible for free/reduced-price lunch scored higher, on average, than eligible students by 23 points at grade 4 and by 28 points at grade 8. At grade 4, the gaps in Boston and Houston were narrower than the nation's. At grade 8, the District of Columbia, Houston, and Los Angeles had narrower score gaps than large central cities and the nation, while Charlotte had a wider gap in the average score than the gap found in large central cities and in the nation.

Eighth-grade students who participated in the NAEP 2003 mathematics assessments, including those in the TUDA, were asked to indicate, from among five options, the highest level of education completed by each parent. The question was not posed to fourth-graders.

The average score for students who indicated that a parent graduated from college was lower in Atlanta, Chicago, Cleveland, the District of Columbia, and Los Angeles than the average score for students in the same parental education category in public schools in large central cities. The average score for students who reported that a parent graduated from college was higher in Charlotte and San Diego than for comparable students in large central cities as a whole. Students in Boston, Houston, and New York City who reported that a parent graduated from college had an average score that was not found to differ statistically from that of their counterparts in large central cities.

NAEP endeavors to assess all students selected in the randomized sampling process, including SD students and students who are classified by their schools as LEP students. Some students who are sampled for participation, however, can be excluded from the sample according to carefully defined criteria. School personnel, guided by the student's Individualized Education Program (IEP), as well as by eligibility for Section 504 services, make decisions regarding inclusion in the assessment of SD students. Based on NAEP's guidelines, they also make the decision regarding inclusion of LEP students. The process includes evaluating the student's capability to participate in the assessment in English, as well as taking into consideration the number of years the student has been receiving instruction in English. The percentage of students excluded from NAEP may vary considerably across states or districts. Comparisons of achievement results across districts should be interpreted with caution if the exclusion rates vary widely.

¹The NAEP achievement levels are as follows. Basic denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade. Proficient represents solid academic performance for each grade assessed. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate to the subject matter. Advanced signifies superior performance. Detailed descriptions of the NAEP mathematics achievement levels can be found on the NAGB web site (http://www.nagb.org/pubs/pubs.html).

²For Charlotte and Los Angeles, statistical comparisons restricted to just the schools in large central cities, as distinct from the whole-district comparisons used here, are available from the online Data Tool on the NAEP web site (http://nces.ed.gov/nationsreportcard/nde). The results of significance tests in this report for these two districts may differ slightly from those found by type of location in the online Data Tool.