Skip to main content
Skip Navigation

NAEP 1994 Geography A First Look :

Findings from the National Assessment of Educational Progress

Revised Edition, October 1995

Authors: Paul L. Williams, Clyde M. Reese, Stephen Lazer, and Sharif Shakrani

Download sections of the report (or the complete report) in a PDF file for viewing and printing.


Chapter 1: Introduction

With the completion of its 1994 assessment program, the National Assessment of Educational Progress (NAEP) concluded its 25th year as the only nationally representative and continuous assessment of what America's students know and can do in various subject areas. In 1994, the NAEP program included geography assessments that were administered to representative samples of public and nonpublic school students at grades 4, 8, and 12. This report is a first look into the results of this assessment, providing summary data only for the major demographic subpopulations in the nation. The forthcoming NAEP 1994 Geography Report Card will give more detailed information about the results presented here. Perhaps more importantly, it will provide a context for understanding the findings as they relate to instructional content; instructional practices; school and teacher characteristics; school conditions; and student background, student activities, and home environment.

The National Assessment of Educational Progress (NAEP)

NAEP is a congressionally mandated survey administered by the National Center for Education Statistics (NCES), U.S. Department of Education. Since 1969, NAEP has reported on the educational achievement of American students and provided accurate and useful information to parents, educators, and policymakers at the national, state, and local levels. NAEP has become an integral part of our nation's evaluation of the condition and progress of education.

Since its beginning, NAEP assessments have been conducted periodically in reading, mathematics, science, writing, history, geography, and other fields. The NAEP 1994 program included assessments in reading, United States history, and geography. (Separate samples were assessed for each subject.)

The NAEP Geography National Sample

The NAEP 1994 geography assessment was based on a national probability sample of public and nonpublic school students enrolled in grades 4, 8, and 12. Approximately 5,500 fourth-grade students, 6,900 eighth-grade students, and 6,200 twelfth-grade students participated in the assessment. Detailed information about the student sample sizes is presented in Table A.1 in Appendix A. The national sample included students attending domestic Department of Defense (DoD) schools and Bureau of Indian Affairs schools.

The NAEP Geography Framework

The NAEP 1994 geography assessment was based on a new blueprint, the Geography Framework for the 1994 National Assessment of Educational Progress.[1] It was developed through a national consensus process involving geographers and educators from around the country.

The assessment, which covered both global and United States topics, focused on three content areas in geography that established a context for investigating students' knowledge of key aspects of geography.

1994 Assessment Framework

1994 Assessment Framework

Table 1 shows the percentage of assessment time to be devoted to each content area specified in the framework. The percentages were constant across the three grades. In addition to guiding assessment construction, these percentages were used to weight the content subscales in the calculation of the composite geography scale used in this report. (A discussion of the content area subscale results will be included in the forthcoming NAEP 1994 Geography Report Card. )

Table 1

In addition to defining the content of the assessment, the NAEP 1994 Geography Framework described the specific cognitive dimensions to be measured. Each assessment task was designed to measure either "knowledge," "understanding," or "applying." Again, the percentages of assessment time devoted to these three cognitive dimensions were established by the framework. The elements of the geography assessment are illustrated in Figure 1.

The framework also incorporated the use of a wide variety of stimulus material such as maps and diagrams. These were used to measure students' ability to interpret and analyze geographic materials.

Finally, the framework indicated that at least 50 percent of testing time should be spent on constructed-response questions that require students to write short (one or two sentences) or extended (a paragraph or more) answers. In the actual assessment, approximately 60 percent of assessment time was devoted to questions of this type.

At each grade level assessed, the NAEP 1994 geography assessment consisted of a set of test booklets, each containing student background questions and cognitive tasks. The background sections asked students to provide information about their characteristics, classroom instruction, and motivation to complete the assessment. The cognitive sections included stimulus materials and associated tasks designed to assess students' geographic knowledge and skills. A complete cognitive section for each of the three grade levels is reproduced in Appendix B. Each section contains a mixture of multiple-choice and constructed-response questions. At each grade level, the assessment was composed of six 25-minute blocks of cognitive questions. At grades 8 and 12, these were supplemented by one 50-minute block. Each assessed student completed a booklet with either two 25-minute blocks or one 50-minute block. The booklets were distributed randomly to students and required about one hour to complete.

Figure 1

Note: Example questions are illustrative only, and are not meant to represent the full array of assessment content.
* Applying = A range of higher-order thinking skills.

The NAEP Geography Scale

Responses to the assessment tasks were analyzed to determine the percentages of students who responded correctly to each of the multiple-choice questions and the percentages attaining each of the possible scores for constructed-response questions. Item response theory (IRT) methods were used to produce within-grade scales that summarize results for each of the three content areas. Each subscale for grade 4 was linked to the corresponding subscale for grade 8. Likewise, each subscale for grade 12 was linked to the corresponding subscale for grade 8. Then, each linked subscale was mapped onto a 0 to 500 scale. These separate subscales were then weighted by the percentages shown in Table 1 to produce a composite NAEP geography scale, which is used in Chapter 2 to present results.

Though NAEP scales appear similar across subjects (e.g., all NAEP scales for the 1994 assessments range from 0 to 500), the reader is cautioned against making any comparisons among subjects. For each subject, unique scales are developed to describe student performance in the particular subject area.

Achievement Levels

In addition to summarizing results using the NAEP geography scale, this report presents data using the geography achievement levels authorized by the NAEP legislation [2] and adopted by the National Assessment Governing Board (NAGB). The achievement levels were based on collective judgments -- gathered from a broadly representative panel of teachers, education specialists, and members of the general public -- about what students should know and be able to do relative to the body of content reflected in the NAEP assessment framework. Three achievement levels were defined for each of the grade levels assessed: Basic, Proficient, and Advanced. The policy definitions of these achievement levels are given in Figure 2. In reporting NAEP results, however, there are effectively four achievement-level categories: the percentages of students at or above each of the levels and the percentage below the Basic (lowest) level.

Figure 2

It should be noted that the setting of achievement levels on the national assessment is relatively new and in transition. Some evaluations have concluded that the percentages of students at certain levels may be underestimated.[3] On the other hand, critiques of those evaluations have found that such conclusions are not supported by the weight of the empirical evidence.[4]

The student achievement levels in this report have been developed carefully and responsibly, and have been subject to refinements and revisions in procedures as new technologies have become available. Upon reviewing of the available information, the Commissioner of NCES has judged that the achievement levels have a developmental status. However, the Commissioner and the National Assessment Governing Board also believe that the achievement levels are useful and valuable in reporting on the educational achievement of students in the United States.

Overview of this Report

The two remaining chapters of this report present selected results in terms of NAEP geography scale and student achievement levels, respectively. Within each of these chapters, findings are presented for the nation, for the regions, and for the major reporting subgroups described below. More detailed descriptions of the reporting subgroups are presented in Appendix C.

  • Race/Ethnicity. Estimates are reported for students' race/ethnicity (self-identified) using the following mutually exclusive categories: White, Black, Hispanic, Asian-American, Pacific Islander, and American Indian (including Alaskan Native).

  • Gender. Estimates are reported separately for males and females.

  • Parents' Education Level. Estimates are reported based on students' reports of the highest level of education attained by at least one of their parents: did not finish high school, graduated from high school, some education after high school, or graduated from college.

  • Public/Nonpublic Schools. Estimates are reported for students attending public schools and nonpublic schools, including Catholic and other nonpublic schools.

This report examines and compares the geography performance of groups of students defined by shared demographic characteristics or responses to background questions (for example, males compared to females). It does not explore the relationships among combinations of these groups (for example, White males compared to Black males).

The means and percentages presented in the report are estimates because they are based on samples rather than the entire population(s). Consequently, the results are subject to a measure of uncertainty, reflected in the standard error of the estimate. Although standard errors are not provided with the estimates presented in this report, a full set of standard errors will be available in the NAEP 1994 Geography Report Card.

The comparisons presented in the report are based on statistical tests that consider both the magnitude of the difference between the group means or percentages and the standard errors of those statistics. Throughout this report, differences between reporting groups are defined as significant when they are significant from a statistical perspective. This means that observed differences are unlikely to be due to chance factors associated with sampling variability. All differences reported are statistically significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant," therefore, is not intended to imply a judgment about the absolute magnitude or educational relevance of the differences. The term is intended to identify statistically dependable population differences as an aid in focusing subsequent dialogue among policymakers, educators, and the public.

This report contains three appendices. Appendix A provides information about sampling. Appendix B contains sample assessment questions. Appendix C includes descriptions of the reporting subgroups. Detailed information about measurement methodology and data analysis techniques will be available in the forthcoming NAEP 1994 Geography Report Card and the NAEP 1994 Technical Report.

Cautions in Interpretations

The reader is cautioned against making simple or causal inferences related to the performance of various subgroups of students or about the effectiveness of public and nonpublic schools. Average performance differences between two groups of students may be due in part to socioeconomic and other factors. For example, differences observed among racial/ethnic subgroups are almost certainly associated with a broad range of socioeconomic and educational factors not discussed in this report and possibly not addressed by the NAEP assessment program. Similarly, differences in performance between public and nonpublic school students may be better understood after accounting for factors such as composition of the student body, parents' education levels, and parental interest.

Endnotes

  1. Geography Framework for the 1994 National Assessment of Educational Progress (Washington, DC: National Assessment Governing Board, U.S. Department of Education, Government Printing Office).

  2. P.L. 103-382. Improving America's Schools Act of 1994.

  3. Education Achievement Standards, NAGB's Approach Yields Misleading Interpretations, United States General Accounting Office Report to Congressional Requestors (Washington, DC: United States General Accounting Office, June 1993.) GAO/PEMD-93-12 Educational Achievement Standards.

    Setting Achievement Levels for the Nation, The Second Report of the National Academy of Education Panel on the Evaluation of the NAEP Trial State Assessment, 1992 Trial State Assessment (Stanford, CA: National Academy of Education, 1993.)

  4. American College Testing, Technical report on setting achievement levels on the 1992 National Assessment of Educational Progress in mathematics, reading, and writing (Washington, DC: National Assessment Governing Board, 1993.)

    Cizek, G., Reactions to National Academy of Education Report (Washington, DC: National Assessment Governing Board, 1993.)

    Kane, M., Comments on the NAE evaluation of the NAGB achievement levels (Washington, DC: National Assessment Governing Board.)


Go back to Table of Contents for NAEP 1994 Geography: A First Look report

Last updated 5 April 2001 (RH)

Go to Top of Page