Evaluating the Performance of Urban Students

One of the leading issues in American education today is how well students in the nation’s large urban areas are performing. Many large urban school districts are located in states in which they are the only such district. When their students are tested using a state test, these districts can be compared with the rest of the state’s population, but not with large urban districts in other states. The National Assessment of Educational Progress (NAEP) is the only assessment system that can provide comparable statistics across state boundaries. To examine the feasibility and value of using NAEP to compare large urban districts in various states, the National Center for Education Statistics (NCES) inaugurated the Trial Urban District Assessment (TUDA). TUDA provides results for fourth- and eighth-graders in selected urban districts using the same NAEP assessment questions administered at the national and state levels. In 2002, TUDA was conducted for the first time, with assessments in reading and writing. Results from 2002 are featured in this issue of the Quarterly. In 2003, TUDA was conducted again, with assessments in reading and mathematics.

Characteristics of TUDA Districts

In 2002, NCES invited five large urban districts to participate in TUDA: New York City Public Schools, Los Angeles Unified, Chicago School District 299, Houston Independent School District, and Atlanta City. In 2003, four more districts were added: Boston School District, Charlotte-Mecklenburg Schools (in North Carolina), Cleveland Municipal School District, and San Diego City Schools. TUDA required NCES to work closely with school district administrators, as well as with school principals and staff. I appreciate the help that the Council of the Great City Schools provided in working with these large school districts. (In addition to results for the nine selected districts, results for the District of Columbia, which has participated in many NAEP state assessments, are included in TUDA reports.)

The nine districts were chosen to provide variation in characteristics such as student population size, demographic and socioeconomic composition, and geographic location. The size of these districts ranged from about 55,000 students in Atlanta to over one million in New York City. Data from the 2003 TUDA in mathematics at grade 4 show the largest proportion of Black students in Atlanta (87 percent), the largest proportion of Hispanic students in Los Angeles (73 percent), and the largest proportion of White students in Charlotte-Mecklenburg (41 percent). The proportion of limited-English-proficient (LEP) fourth-graders ranged from 2 percent in Atlanta to 56 percent in Los Angeles. All of the fourth-graders in Cleveland were eligible for free or reduced-price school lunch—an indicator of poverty status—while 45 percent of those in Charlotte-Mecklenburg were eligible. The large urban districts were distributed across all four regions of the United States. While most of the districts were located entirely within large cities, two districts, Charlotte-Mecklenburg and Los Angeles Unified, also included schools located in suburban areas.

Not only did the large urban districts differ from each other, but they also tended to differ from the nation as a whole and from the states. For example, all of the participating districts had lower proportions of White students than did the nation. And all of the districts except Charlotte-Mecklenburg had considerably higher proportions of students eligible for free or reduced-price school lunch than did the nation. The dramatic differences between the characteristics of a given urban school district and those of the nation, the states, and other districts create special challenges in analyzing and reporting the data.

Challenges in Analyzing and Reporting TUDA Data

Because of differences in characteristics, it can be difficult to make meaningful comparisons between TUDA districts and the nation or the states. To provide a more appropriate frame of reference, NCES also compared the 2002 results for TUDA districts with the average for all students in central cities across the nation. The term “central city” does not mean the central part of a city or the “inner city,” but rather a city that is central. A central city is defined by the Census Bureau as a city of 50,000 people or more that is the largest in its metropolitan area, or can otherwise be regarded as central, taking into account such characteristics as commuting patterns. However, many central cities—for example, Lawton, Oklahoma, or Parkersburg, West Virginia—do not resemble the TUDA districts to any great degree. In 2003, therefore, NCES refined its comparison group for the TUDA districts to include only large central cities (defined as central cities with populations of at least 250,000). About 15 percent of the nation’s students live in large central cities. In terms of student population characteristics, TUDA districts are often more similar to the large central cities than to the nation or the states.

Another way in which NCES addresses the issue of widely varying characteristics is by comparing the performance of individual subgroups in a given TUDA district with the performance of the same subgroups elsewhere. For example, the performance of Black fourth-graders in Atlanta can be compared with that of Black fourth-graders in the nation, in the large central cities, in Georgia, and in other TUDA districts. Similarly, the performance of students eligible for free or reduced-price lunch can be compared with the performance of other eligible students. Such comparisons frequently demonstrate that subgroup members in TUDA districts are performing at or above the national average for their peers (though of course subgroup members in some TUDA districts are performing below this average).

An additional issue concerns differences in the proportions of students that TUDA districts excluded from the assessments, either because those students had disabilities that prevented their participation, or because their knowledge of English was not sufficient for them to participate meaningfully. In the 2003 TUDA assessment in reading, for example, 33 percent of Houston’s fourth-grade sample was identified as LEP and 20 percent of the sample was excluded. (Houston typically assesses the reading skills of LEP students in Spanish, while NAEP assesses reading skills only in English.) In Los Angeles, 56 percent of the sample was identified as LEP, but only 5 percent of the sample was excluded. (In California, state law requires testing of nearly all students.) Variability in the proportion of students excluded should be taken into consideration when interpreting the results and making comparisons.

Available Results and Future Plans for TUDA

One of the primary values of NAEP to the public is the ability to compare performance in various jurisdictions with performance in other jurisdictions and in the nation. While NCES has always administered NAEP assessments to students in urban areas, TUDA for the first time allows comparisons of students in individual large urban districts to students in the nation, in large central cities (taken as a whole), in the states, and in other participating districts. Such comparative data appear in NCES reports on the 2002 and 2003 TUDA, and one can also make one’s own comparisons using the NAEP Data Tool available on the NCES web site (http://nces.ed.gov/nationsreportcard/nde).

I am excited about the opportunity to provide the public with comparative information about student performance in large urban districts. Additional information will be available after TUDA is conducted again in 2005. However, the program retains its trial status. The decision whether to make urban district assessments a permanent part of NAEP will be made only after NCES and the National Assessment Governing Board (NAGB) further evaluate the results.