Estimation Skills, Mathematics-in-Context, and Advanced Skills in Mathematics:
Results from Three Studies of the National Assessment of Educational Progress Mathematics Assessment
November 1999
Authors: Julia H. Mitchell,
Download any section of the full report in a PDF file for viewing and printing.
Chapter 1: Introduction
This report presents information from three special studies conducted as part of the National Assessment of Educational Progress (NAEP) 1996 mathematics assessment. For more than a quarter of a century, NAEP has been the only nationally representative and continuing assessment of what students in the United States know and can do in various academic subjects. Each NAEP assessment is guided by a framework that specifies important learning outcomes in that subject area; the 1996 mathematics framework was an enhancement of the one used for the mathematics assessments in 1990 and 1992. The goal of the new framework was to define a 1996 assessment that would: (1) more adequately reflect current curricular emphases and objectives, and yet (2) maintain a connection with the 1990 and 1992 assessments to measure trends in student performance.[1]
In addition to the main NAEP assessment, NAEP periodically conducts special studies focused on areas of interest to educators and others. Topics for some of these studies arise as a result of how students performed on NAEP; others are generated simply from research questions about teaching, learning, and assessment of student achievement. This report focuses on studies in mathematics; special studies have also been conducted in, for example, reading and writing.
Purpose and Audience for the Report
This report is intended primarily for mathematics educators and others concerned with mathematics education, such as curriculum specialists, teachers, and university faculty in schools of education. The three studies reported here were designed to provide greater detail on how students perform on particular types of mathematics questions. They include: the Estimation Study; the Study of Mathematics-in-Context, which will be referred to as the Theme Study; and the Study of Students Taking Advanced Courses in Mathematics, which will be referred to as the Advanced Study. The Theme Study and the Advanced Study were administered for the first time in 1996. The Estimation Study, on the other hand, had been administered twice before, in 1990 and 1992.
Major Findings of the Report
The first study was designed to explore students' skills in estimation. It was implemented at three grade levels and was the only one of the studies that provided trend information. Findings from the Estimation Study include the following:
-
Although there has been significant improvement in mathematics performance overall
since 1990 at all grade levels, the trend for student performance in estimation over the
6 years since the inception of the Estimation Study in 1990 is less clear.
-
Student performance in Estimation at grades 4 and 12 was stronger in 1996 than
in 1990.
- Student performance in Estimation at grade 8 appears to be level across the 3 years of the assessment.
The Theme Study was administered at three grade levels and was designed to assess problem-solving abilities within contexts that allow students to make connections across mathematics content areas. Findings from the Theme Study include:
-
At the fourth-grade level, with the exception of the first problem, most students
attempted to answer the questions posed, even though large percentages produced
responses that were scored as "incorrect." Although not definitive, this may be
evidence that the thematic context of the block of questions encouraged students'
attention to the task of solving problems, even ones that proved to be difficult for
most students.
-
At grade 8, unlike grade 4, many students did not attempt to answer the more
complex questions that required them to write explanations or apply concepts in
problem settings.
-
The response rate to the Theme questions at grade 12 was somewhere between the
rates observed for grades 4 and 8, with most questions being attempted by at least
90 percent of the students.
-
At all grade levels, students appear to have difficulty with complex multistep problems,
even those that require only simple computational skills at each step of the problem.
-
At all grade levels, many students seemed to lack the mathematical knowledge needed
to solve problems. Other students, however, appeared to understand the underlying
mathematics but provided incorrect or incomplete responses as a result of carelessness,
inexperience in writing out solutions to problems, or confusion over the wording of
the question.
- At all grade levels, no positive relationship was seen between the frequency with which students engaged in writing a few sentences about how to solve a mathematics problem, or writing reports or doing mathematics projects, and student performance on the Theme blocks.
The Advanced Study was administered at grades 8 and 12 and was designed to provide students who were taking or had taken advanced courses in mathematics an opportunity to demonstrate their full mathematical proficiency. Findings from the Advanced Study include:
-
Students participating in the Advanced Study differed from those who did not
qualify for the study in that they tended to come from homes providing a stronger
educational context, both in materials and in level of parental education. In addition,
based on their participation in Title I programs or qualification for the federal
Free/Reduced-Price Lunch program, fewer Advanced Study students appeared to
come from low-income homes.
-
As would be expected, students at both grade levels who met the criterion for inclusion
in the Advanced Study performed substantially better than other students on the main
NAEP mathematics assessment.
-
The results show that Advanced Study questions were quite difficult, even for students
who were taking the more challenging mathematics courses that were prerequisite for
participation in the study. Overall performance, measured by average percentage
correct, was 36 percent at grade 8 and 30 percent at grade 12. At both grade levels,
moreover, most of these students were unable to solve problems that required two or
three successive steps to achieve the desired result.
- At grade 12, students who were currently taking mathematics or who were, or had been, enrolled in an Advanced Placement (AP) mathematics course outperformed students in the study who were not currently taking a mathematics course or who had not taken an AP course in mathematics.
The 1996 NAEP Mathematics Assessment
To provide a context for the special studies that are the focus of this report, the following sections give additional information about the NAEP 1996 mathematics assessment and about the manner in which the design and execution of the special studies relate to the main mathematics assessment.
NAEP Mathematics Framework
The NAEP mathematics framework encompasses three cross-cutting domains: a content domain, a domain of mathematical abilities, and a domain of mathematical power. The content domain has five strands: Number Sense, Properties, and Operations; Measurement; Geometry and Spatial Sense; Data Analysis, Statistics, and Probability; and Algebra and Functions.[2] The domain of mathematical abilities describes the nature of the knowledge or processes that are involved in successfully handling mathematical tasks or problems; it includes Conceptual Understanding, Procedural Knowledge, and Problem Solving. The domain of mathematical power refers to students' ability to reason, to communicate, and to make connections of concepts and skills across mathematics strands, or from mathematics to other curricular areas. Figure 1.1 summarizes the structure of the framework for the NAEP 1996 mathematics assessment.
The development of the questions for the special studies, although guided by the 1996 NAEP mathematics framework, naturally focused on the goal of each individual study. Questions for the Estimation Study cut across the five content strands, but the main intent was to assess estimation skills. Questions for the Theme Study tended to emphasize problem-solving abilities within the context of real-life types of experiences. Finally, questions for the Advanced Study tended to include more content in Algebra and Functions than did questions in the main NAEP assessment.
In addition to cognitive achievement questions, student assessment booklets for the special studies contained blocks of background questions. The background questions asked students to provide information about themselves, their classroom instruction, and their motivation to expend effort on the assessment. Teachers and school administrators of students participating in NAEP also responded to background questionnaires. Teachers provided information about their education, professional careers, curricular practices, and instructional approaches, as well as the resources available to them for teaching mathematics. School administrators answered questions about school policies and practices.
Samples
The NAEP 1996 mathematics assessment was conducted nationally at grades 4, 8, and 12. As mentioned earlier, both the Estimation Study and the Theme Study also were conducted at grades 4, 8, and 12, while the Advanced Study was conducted at grades 8 and 12 only. Students for the Estimation and Theme Studies were selected through the same sampling design as students for the main NAEP assessment and were representative of all U.S. public and nonpublic school students.[3] Students selected for the Advanced Study were representative of students who had taken, or were enrolled in, more advanced mathematics courses. Specifically, to qualify for the Advanced Study, eighth-grade students had to be currently enrolled in, or already have taken, first-year algebra or a more advanced course in mathematics; and twelfth-grade students had to be currently enrolled in, or already have taken, a pre-calculus or pre-calculus-equivalent course or a more advanced course such as calculus.
Following the model of the main NAEP data collection, school administrators of students participating in the special studies were surveyed at all grade levels, but mathematics teachers of participating students only were surveyed at grades 4 and 8. The exception was the Advanced Study, which included surveys of grade 12 mathematics teachers.
Reporting NAEP Results
Student performance on NAEP assessments has been reported using a variety of measures. Results for the main NAEP mathematics assessment are reported using the NAEP composite mathematics scale, which summarizes performance across five separate subscales -- one for each of the five content strands. In addition to the NAEP mathematics scale, results also are reported using the mathematics achievement levels as authorized by the NAEP legislation[4] and as adopted by the National Assessment Governing Board. The achievement levels are performance standards based on collective judgments about what students should be expected to know and to do. Viewing students' performance from this perspective provides some insight into the adequacy of students' knowledge and skills and the extent to which they achieved expected levels of performance. The Board reviewed and adopted the recommended achievement levels derived from the judgments of a broadly representative panel that included teachers, education specialists, and members of the general public.
For each grade tested, the Board has adopted three achievement levels: Basic, Proficient, and Advanced. For reporting purposes; the achievement level cut scores for each grade represent the boundaries between four ranges on the NAEP mathematics scale: below Basic, Basic, Proficient, and Advanced. The generic policy definitions of the achievement levels are shown below in Figure 1.2. The text of the descriptions of expected mathematics performance at each achievement level at each grade can be found in the NAEP 1996 Mathematics Report Card.[5]
The NAEP legislation requires that the achievement levels be used on a developmental basis until the Commissioner of Education Statistics determines, as the result of a Congressionally mandated evaluation by one or more nationally recognized evaluation organizations that the achievement levels are "reasonable, valid, and informative to the public." Upon review of the available information, the Commissioner of Education Statistics agrees with the recent recommendation of the National Academy of Science that caution needs to be exercised in the use of the current achievement levels, since in the opinion of the Academy "... appropriate validity evidence for the cut scores is lacking; and the process has produced unreasonable results."[6] Therefore, the Commissioner concludes that these achievement levels should continue to be considered developmental and should continue to be interpreted and used with caution. The Commissioner and the Governing Board believe that the achievement levels are useful for reporting on trends in the educational achievement of students in the United States.
Reporting Results for the Special Studies
None of the special study assessment questions contributes to the NAEP composite mathematics scale. However, in 1990, the first year the composite scale and its component subscales were used, a separate scale was established that summarized performance on questions used in the Estimation Study. Each scale was constructed separately, and the metrics of the scales are arbitrary. Therefore, although each scale ranges from 0 to 500 across the three grade levels assessed, it is not possible to conclude that a student who performed at level 300 on the estimation scale, for example, and one who performed at level 300 on the main mathematics scale had both mastered the same proportion of their respective content domains.[7] The value of the estimation scale, like the value of the composite scale, is that it allows for trend analysis across years as well as making it possible to report the results using achievement levels.
The results from the Advanced Study and the Theme Study did not lend themselves to either the development of separate proficiency scales or equating to the main NAEP mathematics scales.[8] Consequently, the overall results from the Theme Study and the Advanced Study are reported simply in terms of the percentages of questions that students answered correctly. Student performance on individual items also is highlighted for each of these studies.
Organization of the Report
Each special study is presented in a separate chapter. The second chapter of this report describes the Estimation Study, the third chapter depicts the Theme Study, and the fourth chapter characterizes the Advanced Study. This report also includes two appendices. The first provides additional information on the procedural and technical aspects of these special studies, and the second includes standard error tables for the data presented in the body of the report.
- For a more in-depth description of the 1996 mathematics framework and how it guided the
development of cognitive items, see the following: National Assessment Governing Board (1996).
Mathematics framework for the 1996 National Assessment of Educational Progress. Washington, DC:
Author; Reese, C. M., Miller, K. E., Mazzeo, J., & Dossey, J. A. (1997). NAEP 1996 mathematics
report card for the nation and the states. Washington, DC: National Center for Education Statistics.
- The content strand Number Sense, Properties, and Operations was called Numbers and Operations
in the 1990 and 1992 assessments. The content strand Geometry and Spatial Sense was called Geometry
in the 1990 and 1992 assessments.
- See Appendix A for detailed information on sample selection.
- The National Education Statistics Act of 1994 requires that the National Assessment Governing
Board develop "appropriate student performance levels" for reporting NAEP results.
- Reese, et al., (1997). op. cit.
- Pellegrino, J. W., Jones, L. R., & Mitchell, K. J. (Eds.). (1999). Grading the nation's
report card: Evaluating NAEP and transforming the assessment of educational progress. Committee on
the Evaluation of National and State Assessments of Educational Progress, Board on Testing and
Assessment, Commission on Behavioral and Social Sciences and Education, National Research Council.
(p. 182). Washington, DC: National Academy Press.
- In the initial year of use, each scale was set to have a mean of 250 and a standard
deviation of 50.
- See Appendix A for more detail.
Download any section of the full report in a PDF file for viewing and printing:
Chapter 3 - Study of
Mathematics-in-Context 9,323K
NCES 2000-451 Ordering information
Suggested Citation
U.S. Department of Education. Office of Educational Research and Improvement. National Center for Education
Statistics. Estimation Skills, Mathematics-in-Context, and Advanced Skills in Mathematics, NCES 2000-451, by
J. H. Mitchell, E. F. Hawkins, F. Stancavage, and J. A. Dossey. Washington, DC: 1999.
Last updated 14 March 2001 (RH)
