Dr. Gary W. Phillips
Deputy Commissioner of the National Center for Education Statistics
The Release of the National Assessment of Educational Progress (NAEP) The Nation's Report Card: U.S. Geography 2001
June 21, 2002
In 2001, NCES conducted the National Assessment of Educational Progress in U.S. geography among fourth-, eighth-, and twelfth-grade students. Results for the 2001 assessment are available for the nation only, with no state data. Students in both public and nonpublic schools participated in the assessment.
The NAEP 2001 geography assessment was based on a framework developed by the National Assessment Governing Board (NAGB), which establishes policy for NAEP. Previously, an assessment in geography was conducted in 1994 using the same framework. Therefore, the results for the 2001 assessment can be compared to the 1994 assessment results.
The NAEP 2001 geography assessment provides results for subgroups of students defined by various background and contextual characteristics. This assessment also provides results for a second sample in which testing accommodations were provided to students with special learning needs (students with disabilities and/or limited English proficient students).
The number of students assessed nationally in 2001 totaled almost 25,000 students. The fourth-grade sample included about 7,000 students in 365 schools, while the eighth- and twelfth-grade samples were approximately 9,000 students each in 369 and 374 schools, respectively.
A Cautionary Note on Interpreting the Results
Please note that because NAEP scores are based on samples, there is a margin of error associated with each score. When comparisons are made between scores, or between the percentages of students at the various achievement levels, the differences must be tested to see if they are larger than the margin of error involved-these differences are referred to as "statistically significant." NCES only discusses statistically significant differences. Furthermore, NAEP is not designed to indicate the reasons for these differences. NAEP is like a thermometer. It tells what the temperature is, but doesn't tell why the temperature changes.
The NAEP Geography Framework
The content of the NAEP 2001 geography assessment is based upon the NAEP Geography framework, which is organized around three content areas and three cognitive areas The three content areas are as follows:
The three cognitive areas of geography are identified in the framework as Knowing, Understanding, and Applying. They are defined as follows:
Students answered both multiple-choice and constructed-response questions. Students devoted more than 50 percent of their time to “constructed-response” questions that required at least a brief written response and sometimes several paragraphs to explain their answers. Some construction-response questions also required students to perform such tasks as identifying location points on a map, indicating travel routes, or drawing a map.
Results Reported from the 2001 Geography Assessment
Student performance is reported in two ways: 1) as average scale scores, and 2) as percentages of students performing at or above various achievement levels. An average scale score reflects the overall geography performance of a particular group or subgroup of students. The achievement levels are intended to describe standards for what students should know and be able to do. There are three geography achievement levels established by the National Assessment Governing Board, defined as Basic, Proficient, and Advanced. NAGB believes that all students should be performing at or above the Proficient level.
National Scale Score Results Across Years
The NAEP geography assessment uses a single 500-point composite scale. All scores, regardless of grade, are on a scale that ranges from 0 to 500. In 2001, the average score for fourth- and eighth-grade students was higher than in 1994, while scores for twelfth-grade students did not show a change. Fourth-grade scores rose from 206 in 1994 to 209 in 2001, while eighth-grade scores rose from 260 to 262. Twelfth-grade students recorded an average score of 285 in both 1994 and 2001.
National Scale Score Percentiles Across Years
In addition to presenting average scores, the results are presented for students at various levels of performance, or percentiles. The report indicates performance trends for students scoring at the 10th, 25th, 50th, 75th, and 90th percentiles in each grade. The improvement in performance at the fourth- and eighth-grade levels was seen among the students scoring in the lower percentiles. Fourth-graders at or near the 10th and 25th percentiles showed higher scores, increasing from 146 to 158 and from 179 to 185, respectively. Scores for students at or near the 50th, 75th, and 90th did not change. At the eighth-grade, students at or near the 10th and 25th percentiles also showed a change. Specifically, scores increased from 213 to 217 (10th percentile) and from 237 to 241 (25th percentile). Scores for the twelfth-grade students showed no change at any of the percentiles.
National Achievement Level Results Across Years
At the fourth grade, the percentage of students who scored below Basic fell from 30 percent in 1994 to 26 percent in 2001, while the percentage at or above Basic, which includes the Proficient and Advanced levels, rose from 70 percent to 74 percent. However, neither the percentage of students at or above Proficient (21 percent) nor at Advanced (2 percent) showed an increase.
At grade 8, there were increases in the percentage of students who were at or above Basic (rising from 71 percent to 74 percent). There was a corresponding decline in the percentage of students who were below Basic, from 29 percent to 26 percent. The percentages of students at or above Proficient (30 percent), and at Advanced (4 percent) did not show a change.
At grade 12, 71 percent of students were at or above Basic, 25 percent were at or above Proficient, and 1 percent were Advanced in 2001. The percentages of twelfth-grade students attaining the Basic, Proficient, and Advanced achievement levels did not show change from 1994 to 2001.
National Scale Scores Across Years by Gender
Scores for male and female students did not show increases or declines from 1994 to 2001. The sample size for male and female students is necessarily half the size of the overall sample. This increases the margin of error, and makes it less likely that statistically significant differences will be identified. The average scale score for male fourth-graders was 212, while the score for females was 207. At the eighth grade, the average male score was 264 in 2001, while the average for females was 260. At grade 12, the average score for males was 287 in 2001, and the average score for females was 282. However, while trend changes for males and females were not statistically significant in 2001, male students had higher average scores than female students in all three grades, just as they had in 1994.
National Achievement Level Results Across Years by Gender
At the fourth and twelfth grades, the percentages of both male and female students who were at the Basic achievement level showed increases from 1994 to 2001. The percentage of male fourth-graders at the Basic level increased from 46 percent to 51 percent. The percentage of female fourth-graders at the Basic level increased from 49 percent to 54 percent. The percentage of male twelfth-graders at the Basic level increased from 41 percent to 45 percent. The percentage of female twelfth-graders at the Basic level increased from 45 percent to 48 percent. These were the only changes in achievement level percentages.
National Scale Scores Across Years by Race/Ethnicity
Scores for racial/ethnic subgroups on the geography assessment showed only one change in performance, across all three grades. Fourth-grade Black students showed an increase, from 168 in 1994 to 181 in 2001.
National Achievement Level Results Across Years by Race/Ethnicity
At the fourth grade, both Black and White students showed a change in achievement level percentages from 1994 to 2001. The percentage of Black students at or above the Basic level increased from 34 percent to 44 percent, while the percentage below Basic fell from 66 percent to 56 percent. The percentage of White students at or above the Basic level increased from 81 percent to 87 percent, while the percentage below Basic fell from 19 percent to 13 percent.
At the eighth and twelfth grades, only White students showed a change in achievement level percentages from 1994 to 2001. In the eighth grade, the percentage of White students at or above the Basic level increased from 82 percent to 86 percent, while the percentage below Basic fell from 18 percent to 14 percent. In the twelfth grade, the percentage of White students at the Basic level rose from 46 percent to 51 percent. This was not enough to produce an increase in the percentage at or above Basic.
Racial/Ethnic Gaps in National Scale Scores Across Years
Comparisons of the differences in average scores for White and Black students and White and Hispanic students showed one change from 1994 to 2001. At the fourth grade, the average score gap between White and Black students decreased from 50 points to 40 points. All other differences in average scores did not show a change.
National Scale Scores by Parents' Highest Level of Education
NAEP provides information on student performance according to parents' highest level of education for grades 8 and 12 only. Students are asked to report the highest level of education attained by either parent. Fourth-graders are not asked this question due to the poor reliability of the responses.
At the eighth grade, students who reported that they did not know their parents' level of education showed an increase in average score (from 234 in 1994 to 245 in 2001). However, the average score of students that did not know their parents' level of education was among the lowest.
At the twelfth grade, students who reported that neither of their parents finished high school showed an increase in average score, from 263 in 1994 to 269 in 2001. For both grades, the higher the parental education level reported, the higher the average score attained.
Average Scale Scores by Type of School (Public and Nonpublic)
NAEP collects data for students in both public and nonpublic schools. Results are presented for public and nonpublic schools, where nonpublic schools are divided into the categories of Catholic and “Other” nonpublic schools. In 2001, average scale scores public schools increased at the eighth grade only, rising from 258 in 1994 to 261. Scores for combined nonpublic students, Catholic students, and “Other” nonpublic students did not show a change at any of the three grades. At all three grades, students in nonpublic schools showed higher average performance than those in public schools, a pattern that is visible in achievement level data as well.
In addition to assessing performance in geography, NAEP also collected data from students and teachers about classroom and instructional activities that may be related to performance. Some activities that showed a relationship to student performance are discussed below.
Teacher Preparation: Grade 4
The NAEP data indicate that at the fourth grade, 93 percent of students had teachers who reported a “major, minor, or special emphasis” in elementary education, either as undergraduates or graduates. Students of these teachers attained a higher average score (211) than the 7 percent of students who did not have such teachers (average score, 197).
Geography Skills and Topics: Grades 8 & 12
The NAEP geography assessment asked eighth- and twelfth-grade students a number of questions about the frequency with which they received instruction in a variety of geography skills and topics. Eighth-grade students who said they were taught how to use maps and globes “once or twice a month” had a higher average score (268) than those said they received such instruction “almost every day,” “once or twice a week,” or “never or hardly ever.” Eighth-graders who said that they learned about natural resources once or twice a month also had a higher average score (269) than those who said they studied the topic either more or less frequently. Eighth-grade students who said that they never or hardly ever learned about “countries and cultures” had a lower average score (254) than those who said they studied the topic more frequently.
Twelfth-graders who said they studied how to use maps and globes almost every day had a lower average score (277) than students who said they had less frequent instruction in this skill. The same pattern was true for the study of natural resources (average score, 275). As in the eighth grade, twelfth-graders who said that they never or hardly ever learned about “countries and cultures” had a lower average score (277) than those who said they studied the topic more frequently.
Geography Course Taking: Grades 8 & 12
Eighth-graders were asked, “Did you take or do you expect to take a geography course in the 6th, 7th, or 8th grade?” Those who selected two or three grades had higher average scores (263 and 272) than those who selected fewer or answered “don’t know.”
Twelfth-graders were asked, “Did you take or do you expect to take a geography course in the 9th, 10th, 11th, or 12th grade?” Students who selected “none” or “one” had higher average scores (289 and 288) than those who selected a higher number or chose “don’t know.” The students who reported taking no more than one year of geography may have been following a different academic curriculum than the students who reported taking geography in multiple years.
Use of Computers: Grades 4 & 8
Most fourth- and eighth-graders said they had used a computer to do class work. At grade four, students who said they never used a CD-ROM to look up reference works had a lower average score (205) than students who used CD-ROMs to a “small” or “moderate” extent. Similarly, fourth-graders who said they never retrieved information through the Internet had a lower average score (203) than those who said they did so to a “small” or “moderate” extent. The same patterns were true for eighth-graders as well (low scores of 258 and 255, respectively). Few students reported using computers to a “large” extent (4 percent or less), which generally prohibited comparisons involving these students.
Use of Computers: Grade 12
Most twelfth-graders (74 percent) said they had used a computer to do class work. Students who said they had not used a computer to do research project involving a CD-ROM or the Internet had a lower average score (274) than students who said they had.
Days Absent from School: Grades 4, 8 & 12
[These data aren’t in the Report Card.]
Accommodations Permitted in the 2001 Geography Assessment
In recent years, NAEP has been investigating the impact of permitting testing accommodations for students with disabilities and limited English proficient students. These students are unable to fully participate in the NAEP assessment without accommodations. NAEP only allows accommodations that do not affect the integrity of the assessment. For example, NAEP allowed some students to use a bilingual dictionary for the 2001 U.S. history assessment, but not for the 2000 reading assessment, because NAEP took the position that the reading assessment is intended to assess students' ability to read English. In a few instances, including accommodated students has led to lower average scores. However, NAEP has concluded that providing accommodations has only a minor impact on statistical results overall.
NAEP used two samples in the 2001 geography assessment. One sample was assessed with accommodations permitted for students who needed them to participate and one sample was assessed without accommodations. Average scores for the non-accommodated sample were used to allow comparisons with the 1994 assessment, which did not allow accommodations. There were no significant differences in the average scores between the two sets of results at grades 4 and 12. At grade 8, however, the average score when accommodations were permitted was lower than the average score when accommodations were not permitted. Scores from the two samples appear in the main body of the report card.
Complete results for the Nation's Report Card: U.S. History 2001 are available on the web at: http://nces.ed.gov/nationsreportcard/geography/results/.