Interpreting NAEP Mathematics Results
Overview of the Assessment Overview of the AssessmentNAEP assesses student mathematics performance by administering samples that are representative of the nation's students. The content of the NAEP mathematics assessment is determined by a framework incorporating expert perspectives about mathematics and its measurement. Read more about what the assessment measures, how it was developed, who took the assessment and how the assessment was administered. The 2003 mathematics results presented on the website are based on representative samples of students for the nation and for participating states and other jurisdictions. Approximately 343,000 students from 14,000 schools were assessed. The national results reflect the performance of students attending both public and nonpublic schools, while the state and jurisdiction results reflect only the performance of students attending public schools. Beginning in 2002, the NAEP national sample was obtained by aggregating the samples from each state, rather than by obtaining an independently selected national sample. As a consequence the national sample size increased, and smaller differences between years or between types of student were found to be statistically significant than would have been detected in previous assessments. In keeping with past practice, all statistically significant differences are indicated in the current web results pages. Comparisons are made to results from previous years in which the assessment was administered. In addition to the 2003 results, state and national results are reported from the 1992, 1996, and 2000 assessments at grade 4 and the 1990, 1992, 1996, and 2000 assessments at grade 8. The more recent national results (those from 1996 or later) are based on administration procedures in which testing accommodations were permitted for students with disabilities and for limited-English-proficient students. Accommodations were not permitted in earlier assessments (1990 and 1992). Comparisons between results from 2003 and those from assessment years in which both types of administration procedures were used (1996 and 2000) are discussed based on the results when accommodations were permitted, even though significant differences in results when accommodations were not permitted may be noted in figures and tables. Changes in student performance across years or differences between groups of students in 2003 are discussed only if they have been determined to be statistically significant. Reporting the Assessment—Scale Scores and Achievement LevelsThe results of student performance on the NAEP mathematics assessment are presented on this website in two ways: as average scores on the NAEP mathematics scale and as the percentages of students attaining NAEP mathematics achievement levels. The average scale scores represent how students performed on the assessment. The achievement levels represent how that performance measured up against set expectations for achievement. Thus, the average scale scores represent what students know and can do, while the achievement-level results indicate the degree to which student performance meets expectations of what they should know and be able to do. Average mathematics scale score results are based on the NAEP mathematics scale, which ranges from 0 to 500. The NAEP mathematics assessment is a composite combining separate scales for each of the mathematics content strands: (1) number sense, properties and operations; (2) measurement; (3) geometry and spatial sense; (4) data analysis, statistics and probability; and (5) algebra and functions. Average scale scores are computed for groups, not for individual students. The average scores are based on analyses of the percentages of students who answered each item successfully. While the score ranges at each grade in mathematics are identical, the scale was derived independently at each grade. Therefore, average scale scores across grades cannot be compared. For example, equal scale scores on the grade 4 and grade 8 scales do not imply equal levels of mathematics achievement. Achievement-level results are presented in terms of mathematics achievement levels as adopted by the National Assessment Governing Board (NAGB), and are intended to measure how well students' actual achievement matches the achievement desired of them. For each grade tested, NAGB has adopted three achievement levels: Basic, Proficient, and Advanced. For reporting purposes, the achievement level cut scores are placed on the mathematics scales, resulting in four ranges: below Basic, Basic, Proficient, and Advanced. NAGB established its achievement levels in 1996 based upon the mathematics content framework and a standard-setting process involving a cross section of educators and interested citizens from across the nation who were asked to judge what students should know and be able to do relative to the content set out in the NAEP mathematics framework. As provided by law, NCES has determined that the achievement levels are to be considered on a trial basis and should be interpreted and used with caution. However, both NCES and NAGB believe these performance standards are useful for understanding trends in student achievement. Description of Mathematics Performance by Item Maps for Each GradeThe performance of fourth- and eighth-graders can be illustrated by maps that position question descriptions along the NAEP mathematics scale. The descriptions used on these maps focus on the mathematics skill or knowledge needed to answer the question. For multiple-choice questions, the description indicates the skill or knowledge demonstrated by selection of the correct option; for constructed-response questions, the description takes into account the skill or knowledge specified by the different levels of scoring criteria for that question. Approximately 28 mathematics questions per grade have been selected and placed on an item map for grade 4 and grade 8. Results are EstimatesThe average scores and percentages presented on this website are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented on this web site are available through the NAEP Data Tool. NAEP Reporting GroupsResults are provided for groups of students defined by shared characteristics—gender, race or ethnicity, school's type of location, Title I participation, eligibility for free/reduced-price school lunch, and type of school. Based on participation rate criteria, results are reported for subpopulations only when sufficient numbers of students and adequate school representation are present. The minimum requirement is at least 62 students in a particular subgroup from at least five primary sampling units (PSUs). However, the data for all students, regardless of whether their subgroup was reported separately, were included in computing overall results. Explanations of the reporting groups are presented below. GenderResults are reported separately for males and females. Race/EthnicityIn all NAEP assessments, data about student race/ethnicity is collected from two sources: school records and student self-reports. Before 2002, NAEP used student self-reported race and ethnicity as the source of information reported for race/ethnicity. In 2002, it was decided to change the student race/ethnicity variable highlighted in NAEP reports. Starting in 2002, NAEP reports of race/ethnicity are based on school records; student self-reported race/ethnicity will continue to be reported in the NAEP Data Tool. In order to allow comparisons across years, all mathematics results presented in this report are based on school-reported information for six mutually exclusive racial/ethnic categories: White, Black, Hispanic, Asian/Pacific Islander, American Indian (including Alaska Native), and Other. Students who identified with more than one of the first five categories or had a background other than the ones listed were categorized as Other. Type of LocationResults from the 2003 assessment are reported for students attending schools in three mutually exclusive location types: Central city: The U.S. Census Bureau defines "central city" as the largest city of a Metropolitan Statistical Area (MSA) or a Consolidated Metropolitan Statistical Area (CMSA). An MSA is an area defined by the federal government for the purposes of presenting general-purpose statistics for metropolitan areas. Typically, an MSA contains a city with a population of at least 50,000 and includes its adjacent areas. An MSA becomes a CMSA if it meets the requirements to qualify as a metropolitan statistical area, has a population of 1,000,000 or more, its component parts are recognized as primary metropolitan statistical areas, and local opinion favors the designation." Urban fringe/large town: The urban fringe category includes any incorporated place, census-designated place, or non-place territory within a CMSA or MSA of a large or mid-sized city that is defined as urban by the U.S. Census Bureau, but which does not qualify as central city. A large town is defined as a place outside a CMSA or MSA with a population greater than or equal to 25,000. Rural/small town: Rural includes all places and areas with populations of less than 2,500 that are classified as rural by the U.S. Census Bureau. A small town is defined as a place outside a CMSA or MSA with a population of less than 25,000, but greater than or equal to 2,500. Starting in 2000, NAEP revised the method for determining the type of location, because new methods are used by NCES to identify the type of location assigned to each school in the Common Core of Data (CCD). The new methods were put into place by NCES in order to improve the quality of the assignments, and they take into account more information about the exact physical location of the school. The results by type of location in 2003 are therefore compared only to those from 2000. To provide a context for the data collected in the 2003 Trial Urban District Assessment in mathematics (TUDA), results are presented for students attending public schools in the nation as a whole, as well as for public schools located in large central cities across the nation. Large central cities are defined by NCES as cities with a population of 250,000 or more. Title I ParticipationBased on available school records, students were classified either as currently participating in a Title I program, receiving Title I services, or not participating. The classification applies only to the school year when the assessment was administered (i.e., the 2002–03 school year) and is not based on participation in previous years. If the school does not offer any Title I programs or services, all students in that school are classified as not participating. Eligibility for Free/Reduced-Price School LunchAs part of the Department of Agriculture's National School Lunch Program, schools can receive cash subsidies and donated commodities in turn for offering free or reduced-price lunches to eligible children. Based on available school records, students were classified as either currently eligible for the free/reduced-price school lunch or not eligible. Eligibility for free and reduced-price lunches is determined by students' family income in relation to the federally established poverty level. Students whose family income is at or below 130 percent of the poverty level qualify to receive free lunch, and students whose family income is between 130 percent and 185 percent of the poverty level qualify to receive reduced-price lunch. The classification applies only to the school year when the assessment was administered (i.e., the 2002–03 school year) and is not based on eligibility in previous years. If school records were not available, the student was classified as "Information not available." If the school did not participate in the program, all students in that school were classified as "Information not available." Type of SchoolResults are reported by the type of school that the student attends—public or nonpublic. Nonpublic schools include Catholic and other private schools. Because they are funded by federal authorities, not state/local governments, Bureau of Indian Affairs (BIA) schools, Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS), and Department of Defense Dependents Schools (Overseas) are not included in either the public or nonpublic categories; they are included in the overall national results. Exclusion Rates and Assessment ResultsAll 50 states and 3 other jurisdictions participated in the 2003 mathematics assessment. To ensure that the samples in each state are representative, NAEP has established policies and procedures to maximize the inclusion of all students in the assessment. Every effort is made to ensure that all selected students who are capable of participating meaningfully in the assessment are assessed. While some students with disabilities and/or limited-English-proficient (SD and/or LEP) students can be assessed without any special procedures, others require accommodations to participate in NAEP. Still other SD and/or LEP students selected by NAEP may not be able to participate. Local school authorities determine whether SD/LEP students require accommodations or shall be excluded because they cannot be assessed. The percentage of SD and/or LEP students who are excluded from NAEP assessments varies from one jurisdiction to another and within a jurisdiction over time. If excluded students are less proficient in mathematics, variations in exclusion rates could have an impact on average mathematics scores or score gains within jurisdictions. NCES is currently sponsoring ongoing research on the potential impact of changes in exclusion rates on changes in average mathematics performance. The preliminary findings from the research suggest that the potential impact on mathematics scores is minimal. Statistical SignificanceThe differences between scale scores and between percentages discussed in the results on this website take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics. Throughout the results, differences between scores or between percentages are discussed only when they are significant from a statistical perspective. All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policy makers, educators, and the public. Cautions in InterpretationsUsers of this website are cautioned against interpreting NAEP results as implying causal relations. Inferences related to subgroup performance or to the effectiveness of public and nonpublic schools, for example, should take into consideration the many socioeconomic and educational factors that may also impact performance. Beginning in 2002, the NAEP national sample was obtained by aggregating the samples from each state, rather than by obtaining an independently selected national sample. As a consequence the national sample size increased, and smaller differences between years or between types of student were found to be statistically significant than would have been detected in previous assessments. The NAEP mathematics scale makes it possible to examine relationships between students' performance and various background factors measured by NAEP. However, a relationship that exists between achievement and another variable does not reveal its underlying cause, which may be influenced by a number of other variables. Similarly, the assessments do not reflect the influence of unmeasured variables. The results are most useful when they are considered in combination with other knowledge about the student population and the educational system, such as trends in instruction, changes in the school-age population, and societal demands and expectations. Return to main results.
|
