Commissioner, National Center for Education Statistics
National Assessment of Educational Progress 2005 Science Trial Urban District (TUDA) Results
November 15, 2006
Good morning. My name is Mark Schneider, Commissioner of the National Center for Education Statistics. I am here today to share with you the results of the 2005 Trial Urban District Assessment in Science—the TUDA Science Report Card.
The NAEP trial urban district assessments, which we call TUDA, are a collaboration involving the National Center for Education Statistics, the National Assessment Governing Board, and the Council of Great City Schools.
The TUDA assessments were designed to explore the feasibility of using NAEP to measure performance of public school students at the district level on a common scale. Because the assessments are the same for the nation, the states, and the urban districts, NAEP serves as a common yardstick for comparison.
Participation in TUDA is voluntary. In 2005, 10 districts from around the country participated.
The 10 districts participating in the Science TUDA range in size from New York, with almost a million students, to Atlanta, with about 50,000. In between are Los Angeles, Chicago, Houston, San Diego, Charlotte (North Carolina), Austin, Cleveland, and Boston.
The District of Columbia participated in previous TUDA assessments, but because of its small population size could not participate in the Science TUDA Assessment.
Students in the TUDA districts differ in many ways from the nation as a whole and even from the rest of the large central cities in the United States.
One way the TUDA districts differ is in the percentage of non-white students. We compare the percentages of non-white students in the fourth grade in the public schools for each of our districts with the percentage of non-white public school students for the nation as a whole and the average percentage for large central city schools nationwide. NCES defines "large central city" as a central city with a population at or above 250,000.
All 10 urban school districts in this study have percentages of non-white public school students that are higher than the percentage nationwide and, in many districts, higher than that in large central cities as well. (When we compare an individual district to the large central city average, we remove that district’s contribution to the average, so that we aren’t comparing a district to itself.)
We see a similar pattern when we compare the percentages of low-income students in the fourth grade. NAEP defines low-income students as those who are eligible for the National School Lunch Program.
Again, in almost all cases, the percentages of low-income students in the TUDA districts are higher than in the nation, and, for most districts, higher than in the large central cities as well. At the eighth grade, the patterns are similar.
Because this is the first TUDA science assessment, we cannot make comparisons to prior years.
We assessed approximately one thousand public school students per grade in smaller districts, such as Atlanta and Cleveland and around two thousand students in larger districts such as New York, Houston and Los Angeles.
In the NAEP science assessment, student performance is presented on separate 0–300 scales for each grade. NAEP scale scores tell us what students know and what they can do.
Student performance results are also reported according to three achievement levels established by the National Assessment Governing Board, which sets policy for NAEP. These levels set standards for what students should know and be able to do. As you see, we have three achievement levels—Basic, defined as “partial mastery of the knowledge and skills that are fundamental for proficient work;” Proficient, defined as “demonstrating competency over challenging subject matter;” and Advanced, which signifies “superior performance.” The Governing Board believes that all students should perform at the Proficient level.
Students are assessed in three fields of science—Earth science, physical science, and life science. More detailed information about these three fields, and other aspects of the assessment, are available online. Student performance by district for the individual fields of science is available as well. These scores show patterns similar to the overall average scores presented here today and in the Report Card.
Now we’ll put fourth-grade performance under a microscope, looking at it in more detail, but first I want to make a few remarks about comparisons of student performance.
All NAEP results—whether scale scores or achievement level percentages—are based on samples. This means there is a margin of error associated with every score and percentage. Differences in scores and percentages must be tested to ensure that they are statistically significant; that is, that they are larger than the combined margin of error.
In this presentation, I will be comparing the results for the ten urban districts to those for the nation and to the average for students in large central cities nationally. As I described a moment ago, many of these ten districts have higher percentages of traditionally lower-scoring groups, such as minorities and students from low-income backgrounds, than the nation. And you will see that several of these cities scored above the nation when comparing students of the same race/ethnicity.
That doesn’t change a basic fact: at grade four, all of these districts, except for Austin, scored below the nation in science, a subject fundamentally important to the future competitiveness of the United States.
So how did these districts do? Here are the national and large central city numbers.
Seven districts scored as well as, or better than, students in large central cities. These were Austin, Charlotte, Houston, San Diego, New York City, Atlanta, and Boston.
Three districts—Cleveland, Chicago, and Los Angeles—scored below large central cities.
We also compare the percentage of fourth-graders who were at or above Basic and at or above Proficient with the large central city percentages.
Two districts—Austin and Charlotte—had percentages at or above Basic that were higher than the large central city percentage.
In three districts—San Diego, Houston, and New York City—the difference was not significant.
In the remaining five districts, the percentage at or above Basic was lower than the large central city percentage. These were Boston, Atlanta, Cleveland, Chicago, and Los Angeles.
The Governing Board’s goal is that all students should perform at or above Proficient.
The percent of students at this level in the TUDA districts ranges from 9 percent to 25 percent, compared with 27 percent for the nation and 15 percent in large central cities nationwide.
The percentages at or above Proficient in two districts, Austin and Charlotte, were higher than in large central cities. Four districts had lower percentages than central cities nationally of students at or above this level—Boston, Chicago, Cleveland, and Los Angeles.
Comparing districts to one another, Austin and Charlotte were not significantly different from one another and scored higher than the remaining eight districts. Houston and San Diego were lower than Austin and Charlotte but not significantly different from one another. Chicago and Los Angeles scored lower than any district except Cleveland and each other.
Next, we’re going to look at how districts’ performance changes when we consider scores of low-income students only, because the districts serve so many of these students.
As we saw earlier, the percentage of low-income fourth-graders in 9 of the 10 TUDA districts exceeds the national percentage. The percentage of these students also varies substantially among districts, from 46 percent in Charlotte to 100 percent in Cleveland. In 9 of the 10 districts, over 60 percent of fourth-graders are low-income.
On average, these students tend to have lower scores than other students, and some of the variation in scores that we see when comparing districts to one another is due to the variation in percentages of low-income students.
When we compare scores for low-income students only, there are fewer significant differences in scores among districts than when we compare them across their entire student population. Austin scores higher than only four districts—San Diego, Atlanta, Chicago, and Los Angeles. Boston, Houston, and New York City score higher than three other districts—Atlanta, Chicago, and Los Angeles. Charlotte scores ahead of Chicago and Los Angeles, and Cleveland and San Diego score ahead of Los Angeles.
Now I’ll compare the differences, or gaps, in performance in each district and performance in the nation when all students are considered, to these gaps for only the low-income students.
All of the TUDA districts except for Austin scored below the national average, both for all students and for low-income students only. When we compare the national with the individual districts, however, the gaps tend to be narrower if only low-income students are compared.
The gaps in average scores between the nation and each district when all students are included range from 2 points for Austin to 24 points for Chicago and Los Angeles. When only low-income students are considered, these nation-district gaps have a narrower range, from 0 for Austin to 14 points in Los Angeles. The larger gaps we see between most districts and the nation for overall scores, as compared with those for low-income students, are due in part to the fact that in 9 of the 10 districts the percentage of low-income students is larger than in the nation as a whole.
Another way of examining the relative performance of these districts is in terms of the percentage of students in each district that perform as well as or better than students nationwide. Here, I will compare these average percentiles for the nation, central cities, and each district for all students and by race/ethnicity.
The national average score for all public school fourth-graders in science places them at the 47 th percentile, while the large central city average is at the 31 st percentile. The rankings of the 10 districts overall range from Austin at the 45 th percentile to Chicago and Los Angeles at the 22 nd
When we look at student performance within the three major racial-ethnic groups, we see that the percentile rankings of the districts shift and that some districts rank higher than the nation. For example, compared with all students nationally, White students in all of the districts except Cleveland rank above the 50 th percentile, and Atlanta, Austin, and Houston rank above the 75 th percentile, while the national average for White students places them at the 62 nd percentile.
In contrast, Black students in all the districts ranked below the 50 th percentile. Again, some districts are above the national average for Black students. The rank order of the districts shifts as well.
Hispanic students in all 10 districts ranked below the 50 th percentile. As in the case of Black students, several districts are above the national average for Hispanic students, and the rank order for the districts changes once more.
In some districts all three of these racial/ethnic groups perform well compared to the other districts. For other districts, the comparative position of their race/ethnic groups is not as consistent.
I will now examine the gaps in performance between White and Black students in these districts.
White students scored higher than Black students in all 10 districts. In three districts—Boston, Cleveland, and New York City— the gap is smaller than the large central city gap of 37 points.
In Atlanta and Houston, two cities with high-performing White students, the gap is larger than that for large central cities nationally.
The White-Black gaps for the remaining five districts were not significantly different from the large central city gap.
Now we’ll look at the White–Hispanic score gap.
In Cleveland the gap was smaller than the 33-point gap for large central cities, due to the relatively low average score for White students.
In Houston the gap was larger than the large central city gap, due to the relatively high average score for White students in that district.
In the remaining districts, the gap was not significantly different from the large central city gap. In Atlanta, the Hispanic student population at fourth grade is so small that we could not obtain a representative sample, so we did not report an average score for those students.
We asked students several different kinds of questions on the assessment—both multiple-choice and “constructed-response,” which require a written answer. We also asked half the students to perform real experiments.
The example presented is a short constructed-response question—a physics question—that asked students which of two cups of water of equal size would experience the greater increase in the height of water when a ball was placed in the cup. One ball was larger than the other. Students were also required to explain their answers.
A student who chose the cup with the larger ball and gave a correct explanation—that the larger ball would take up more space—received a “Complete” score.
Nationally, 62 percent of students gave a “Complete” answer, while for large central city students, the percentage was 53 percent. For the 10 districts, the range was from 46 percent in Los Angeles to 62 percent in Charlotte.
We have many more sample questions and student responses available on the NAEP website. We make these questions and other analytic tools available to help teachers and school leaders understand where their students may need additional help to do better.
Now we’ll look at the performance of eighth-grade students.
As at grade 4, you will see that when comparing 8 th-grade students within the low-income group, several of these cities performed above their peers in the nation as a whole. Nevertheless, when all students are considered together, at grade 8 all 10 districts performed below the national average.
Austin, Charlotte, and San Diego had scores that were higher than the large central city average nationally. In three districts—Boston, Houston, and New York City—there was no significant difference, and the remaining four were lower than the central city average.
But again, when all students are considered together, at grade 8 all 10 districts performed below the national average.
When we look at the percentages of eighth-graders who were at or above Basic in science for the districts, we see that three—Austin, Charlotte, and San Diego had percentages that were higher than the large central city percentage.
In two districts—Boston and New York City—the difference was not significantly different from the central city percentage.
In the remaining five districts—Houston, Chicago, Cleveland, Los Angeles, and Atlanta-- the percentage at or above Basic was lower than the large central city percentage.
Between 5 percent in Cleveland and 7% in Atlanta up to 27 percent in Austin performed at or above Proficient, which, as I mentioned earlier, is a Governing Board goal. This compares to 27 percent of students nationally and 16 percent in the nation’s large central cities that met this standard.
Looking at cross-district comparisons for all students, we can see that average scores for Austin and Charlotte were higher than the large central city average and higher than the averages for the remaining eight districts. San Diego was higher than the large central city average, lower than Austin and Charlotte, and higher than the remaining seven districts. Boston, Houston, and New York City showed no significant difference when compared to large central cities, were lower than Austin, Charlotte, and San Diego, not significantly different from one another, and higher than the remaining four cities.
Chicago, Cleveland, and Los Angeles were lower than large central cities and the preceding six districts, not significantly different from one another, and higher than Atlanta. All 10 districts were below the national average.
In the previous section, when we looked at fourth-grade performance, we compared the percentile rankings of the districts overall with percentile rankings for the three major racial/ethnic groups. For the eighth grade, we will look at percentile ranking overall and for low-income students only, using eligibility for the National School Lunch Program as the measure of low income.
The nation as a whole performed at the 47 th percentile and large central city students nationally performed at the 31 st percentile. Although we are using percentiles rather than scale scores, the ranking here is the same order that we saw in the cross-district comparison—with Austin, Charlotte, and San Diego below the nation but above the large central cities, for example.
The nation’s low-income student performance falls at the 30 th percentile, and that for large central cities at the 23 rd percentile. Some of the districts shift rank compared to their rankings when all students are included, and all the districts except Atlanta are within 4 percentile points of the large central city average.
We can also compare the gaps between scores for higher- and lower-income students in the 10 districts with the large central city gap.
Higher-income students are defined as those who are not eligible for the National School Lunch Program, while the lower-income students are eligible.
In two districts, Austin and Charlotte, the gap is larger than the 28-point large central city gap.
In the remaining districts, there is no significant difference from the large central cities in the size of this gap.
In Cleveland, all of the students have been declared eligible for the National School Lunch Program, and thus we can’t make this comparison.
One of the questions that eighth-graders answered on the assessment is a multiple-choice question from the field of Life Science, in which students were shown a drawing of a cell and asked them to identify the part of a cell that contains most of its genetic material. The answer is the nucleus.
Nationally, 52 percent of students answered this question correctly, while for large central cities it was 44 percent. For the 10 districts, the percentage ranged from 36 percent in Cleveland to 54 percent in Austin.
Complete information on the 2005 TUDA Science Assessment is available on the Internet at http://nationsreportcard.gov, including the full text of the Report Card, background on NAEP, sample questions, and an online data tool that allows you to do extensive additional comparisons on your own.
In conclusion, I want to thank everyone who worked on the Science Trial Urban District Assessment—the Council of Great City Schools, in particular—and especially the students, the teachers, the schools, and the districts that took part. With their participation, they are helping to provide the best overview of the nation’s academic progress, and in turn, helping educators and policymakers to improve education for all students.
Commissioner Mark Schneider's Powerpoint Presentation:
The Nationís Report Card: Trial Urban District Assessment: Science 2005 (2.7 MB)
Please join Associate Commissioner Peggy Carr at 3:00 pm on Wednesday November 15 for a live chat about the results.