• Background
• Establishing a Benchmark
• Promoting Improvement in Big-City Schools
• References

Results from the 2002 Trial Urban District Assessment (TUDA) in reading and writing were released last summer, presenting the nation with its first look at how some of its largest urban public school systems fared on the National Assessment of Educational Progress (NAEP). Atlanta, Chicago, Houston, Los Angeles, and New York City volunteered for the district-level testing after the Council of the Great City Schools approached the National Assessment Governing Board in the fall of 2000 with a proposal to allow cities to get NAEP results similar to those that participating states have been receiving for many years. Results from the District of Columbia, which participates in NAEP’s state-level assessments, were analyzed along with those from the other cities.

The bid for district-level results was a surprise to many, coming as it did from the nation’s big-city school systems, but the Council made it for a number of reasons:

• First, the nation’s urban school systems wanted to make it crystal clear once again that they are fully committed to the highest academic standards for their students.
• Second, they wanted a way of gauging their progress and evaluating their reforms in ways that the nation’s current state-by-state testing system does not allow.
• Third, they wanted to be able to compare themselves against other school systems facing many of the same challenges.

There were realistic expectations, of course, about what the initial results would show. But the cities were determined to establish a benchmark by which they could measure their progress. That benchmark was set in the 2002 reading and writing TUDA.

The 2002 TUDA results did not present many major surprises. Reading performance in the participating cities was below the national public school average (table 1). The percentage of urban fourth-graders scoring at or above the Basic level of achievement ranged from 48 percent in Houston to 31 percent in the District of Columbia, compared with 62 percent nationally. The percentage of eighth-graders scoring at or above the Basic level ranged from 62 percent in Chicago to 42 percent in Atlanta, compared with 74 percent nationally. (Data on eighth-graders in New York City were not available.)

Writing results indicated that the cities did somewhat better, with two districts having average fourth-grade scores that were statistically indistinguishable from the national average (table 2). The percentage of fourth-graders scoring at or above Basic ranged from 85 percent in New York City to 73 percent in the District of Columbia, compared with 85 percent nationally. The percentage of eighth-graders scoring at or above Basic ranged from 74 percent in Houston to 64 percent in Los Angeles, compared with 84 percent nationally. (Again, data on eighth-graders in New York City were not available.)

Background data collected as part of the assessment also showed how different demographically the TUDA cities are from the rest of the nation. Tables 1 and 2 indicate that the majority of students in these cities, unlike the nation as a whole, were eligible for free or reduced-price school lunch (intended for children near or below the poverty line). The proportion of limited-English-proficient (LEP) students in some of the cities was also considerably higher than the national average. Finally, each of the TUDA cities had enrollments that were predominantly Black or Hispanic, a situation that is significantly different from national averages (not shown in tables 1 and 2).

The initial TUDA results also provided data on achievement by race/ethnicity, eligibility for free or reduced-price school lunch, and other student characteristics, giving analysts and practitioners a better sense of how the performance of subgroups in the cities compares with that of the same subgroups nationally.

The reading results, for instance, indicated that White students, in general, scored at about the same level in TUDA cities as White students nationally. Black students and students eligible for free or reduced-price lunch, on the other hand, generally had lower average scores in TUDA cities than nationally. And Hispanic students in TUDA cities had average scores about the same as or lower than the national average for Hispanic students.

In a number of cities, however, Black, Hispanic, and poor students had reading scores that equaled those of their racial/ethnic and income peers nationally. For example, Black fourth- and eighth-graders in Houston had average scores that did not differ significantly from those of their Black peers nationally. The same was true of Black fourth-graders in New York City and Black eighth-graders in Chicago. At both the fourth and the eighth grades, Hispanic students in three TUDA cities scored about the same as their Hispanic peers nationally. At the fourth grade in New York City and Houston, and at the eighth grade in Chicago, students eligible for free or reduced-price school lunch posted average scores that were statistically indistinguishable from those of eligible students nationally.

A number of interesting results emerged from the writing assessment as well. White students in TUDA cities generally posted average scores that were about the same as or above the national average for White students. Black fourth- and eighth-graders in Houston and Black fourth-graders in New York City outscored their Black peers nationally, while Black students in other TUDA cities scored lower than or about the same as Black students nationally. In almost all cases, the average scores of Hispanic students in TUDA cities were about the same as the average for Hispanic students nationally. Students eligible for free or reduced-price lunch often scored lower in TUDA cities than nationally. In three cities, however, scores of eligible fourth-graders were about the same as the national average for eligible fourth-graders.

The 2002 TUDA results provided fresh data on the size of the achievement gaps within and across the participating cities and how they stack up to gaps nationwide. In some cities, gaps between the achievement of certain subgroups (e.g., White students and Black students) differed from the gaps in the nation as a whole. In other cities, the gaps reflected national averages. At grade 4, for example, the results showed that gaps between the average reading and writing scores of White and Black students in Atlanta and the District of Columbia were wider than the national gaps. But in Houston, Los Angeles, and New York City, the fourth-grade reading and writing gaps between Black and White students were about the same as those nationally.

TUDA results are important to cities and to others interested in the reform and improvement of big-city schools. They provide much-needed data on urban school performance that allows comparisons in ways that the nation’s state-by-state assessment system cannot (Council of the Great City Schools 2003). TUDA will also provide valuable insights into the performance of big-city schools over time.*

But the promise of TUDA stretches well beyond its ability to determine whether big-city schools are improving. Its real importance rests in its power to shed light eventually on which reforms are working and which ones are not. TUDA presents the cities and researchers interested in analyzing patterns in urban schools with a chance to establish a long-term, informal field experiment where each city’s reforms can be gauged against others using a common benchmark.

This kind of research is not commonly done at present, however, in part because there is no common metric on which cities can be measured and in part because it is hard to do. The overwhelming majority of research on urban education is devoted to efforts to turn around individual schools or pockets of schools; efforts to assess the effects of discrete programs or program strategies; and efforts to measure student demographic and background characteristics and the effects of these characteristics on performance.

Some observers have argued that the way to bring about wide-scale improvement in urban education is to take what has been learned about each individual school or program and apply it systemwide. This approach has had limited success, however, because it does not help school practitioners sort through the numerous programs or leverage systemwide change. Very few reforms or research studies, in fact, have focused on how one improves an individual system or how one accelerates improvements across many cities.

Public impatience and No Child Left Behind will no longer allow the nation’s major city school districts to improve themselves using a school-by-school strategy. Time will not allow such a piecemeal approach. Instead, the public and the new law are demanding improvements in city schools writ large. And city school leaders, for their part, need better research on strategies that will help improve student performance systemwide, not just in some schools or for some students.

The lack of solid research on how to turn around public school systems in big cities has been frustrating both for the urban school districts themselves and for others. There has been little way, for instance, to determine which big-city school systems were improving student achievement the most. Consequently, there has been little way to determine for sure what those faster improving school districts were doing to get their gains that others were not doing (Snipes, Doolittle, and Herlihy 2002).

If we had comparable data to tell us, for example, that Black students in Atlanta were improving faster in reading than Black students in other cities, then we would know where to look for promising practices about how to accelerate the reading skills of Black students. If city school systems with varying governance structures were improving at about the same rates, we might conclude that governance structure was not the critical ingredient in the gains. If academic progress in one set of cities extended beyond 3 years, then we might be able to say something about how to sustain the effects of reform. If cities of similar composition were making progress in some states but not in others, then we might be able to ask better questions about the role of states in boosting urban performance. If faster improving cities were using a single reading curriculum and slower moving cities were relying on individual-school reform models, then urban school leaders might have a better sense about how to structure their instructional programs. If faster improving cities had spending levels that were below their respective statewide averages, then researchers and advocates might have a better understanding about the role of money in improving urban schools.

Asking questions of this kind requires a more institutional, systemic, and organizational outlook about the behavior of urban school districts than many in the reform movement or the research community have hitherto shown. There are only a handful of people nationwide, in fact, who think in terms like these. Yet, answers to these and similar questions are critical if urban education is to give the public the results that it wants. Ultimately, urban school leaders will never be able to get gains “at scale” unless they think at scale. TUDA allows that to begin happening.

TUDA will not answer questions about improving big-city school systems on its own, of course. There are too many moving gears in urban schools for us to be certain about anything. But TUDA gives urban educators and others another tool with which to ask better questions.

The 2002 TUDA results are only a starting point. Over time and with a larger pool of participating cities, critical questions about how to leverage improvements in big-city school districts can be answered and acted upon. Urban school leaders are determined to raise student performance in their communities. TUDA will help let them know if they are succeeding. More importantly, TUDA can help them succeed faster. It is an experiment worth pursuing.

Council of the Great City Schools. (2003). Beating the Odds III: A City-by-City Analysis of Student Performance and Achievement Gaps on State Assessments. Washington, DC: Author.

Lutkus, A.D., and Weiner, A.W. (2003a). The Nation’s Report Card: Trial Urban District Assessment, Reading Highlights 2003 (NCES 2004–459). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Lutkus, A.D., and Weiner, A.W. (2003b). The Nation’s Report Card: Trial Urban District Assessment, Mathematics Highlights 2003 (NCES 2004–458). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

Snipes, J., Doolittle, F., and Herlihy, C. (2002). Foundations for Success: Case Studies of How Urban School Systems Improve Student Achievement. Washington, DC: Council of the Great City Schools.

* After this commentary was written, results from the 2003 TUDA were released (Lutkus and Weiner 2003a and 2003b). Reading results for 2003 are now available for the cities that participated in the 2002 TUDA, as well as for four additional ciites.