NAEP - Long-term Trend: Interpret Results

Interpreting NAEP Long-Term Trend Results

Overview of the Assessment
Reporting the Assessment—Scale Scores and Performance Levels
Results Are Estimates
NAEP Reporting Groups
Statistical Significance
Exclusion Rates and Assessment Results
Cautions in Interpretations

Overview of the Assessment

One of the primary objectives of NAEP is to track trends in student performance over time. The NAEP long-term trend assessments in reading and mathematics were administered throughout the nation in the 2003–2004 school year to students aged 9, 13, and 17. Because the long-term trend program uses substantially the same assessments decade after decade, it has been possible to chart educational progress since 1971 in reading and 1973 in mathematics. Measuring trends in student achievement, or change over time, requires the precise replication of past procedures. Therefore, the long-term trend instruments do not evolve based on changes in curricula or in educational practices; in this way, the long-term trend assessments differ from main NAEP. It is not possible to compare results from national or state main NAEP with those of the long-term trend assessment.

For the 2004 administration of the long-term trend assessment in reading and mathematics, several changes were made to the assessment design. Any time changes are made in a long-term trend assessment, studies are required to ensure that the results can continue to be reported on the same trend line—that is, that they are validly comparable to earlier results. Analyses were needed to ensure that the 2004 results under the new design were comparable to the results from 1971 through 1999, under the design that existed earlier. Therefore, two assessments were conducted in 2004. The modified assessment used the new design, and the "bridge" assessment replicated the former design. Comparisons of the results can then detect any shifts in results due to changes in test design. The bridge assessment links the old assessments to the new one. The results described on this website present the results from the bridge assessment used to maintain trend lines between 1971 and 2004.

NAEP assesses student performance by administering samples that are representative of the nation's students. The content of the NAEP long-term trend assessment is determined by a set of objectives incorporating expert perspectives about the measurement of reading and mathematics. Read how the long-term trend assessment was developed, who took the assessment, and how the assessment was administered. Read more about what the long-term trend reading and long-term trend mathematics assessments measure.

The results presented on the website are based on representative samples of students for the nation. Approximately 24,100 students took the modified reading assessment, while about 14,000 took the bridge reading assessment. Approximately 22,400 students took the modified mathematics assessment, while about 14,700 took the bridge mathematics assessment. Although students attending both public and nonpublic schools were sampled, response rates for nonpublic schools selected for participation failed to reach the necessary threshold for reporting separately; therefore, only results for the total sample and public schools are reported.

Reporting the Assessment—Scale Scores and Performance Levels

The results of student performance on the long-term trend assessment are presented on this website in three ways: as average scores, scores at selected percentiles, and percentages of students performing at or above each performance level. Student performance in each subject area is summarized as an average score on a 0-to-500 scale. For each year in which the assessments were administered, achievement in a particular subject area is described for a group of students by their average scale score. Trends in student achievement are determined by examining the average scale scores attained by students in the current assessment year and comparing them to the average scale scores in other assessment years. While the score ranges in both subjects are identical, the scale was derived independently for each subject. Therefore average scale scores between subjects cannot be compared.

Student performance is also described in terms of the percentages of students attaining specific levels of performance. These performance levels correspond to five points on the reading and mathematics scales: 150, 200, 250, 300, and 350. For each subject area, the performance levels from lowest to highest are associated with increasingly advanced skills and knowledge. Examining the percentages of students in each year who attained each performance level provides additional insights into student achievement. Read more about the long-term trend performance levels.

Results are Estimates

The average scores and percentages presented on this website are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented on this website are available in the summary data tables.

NAEP Reporting Groups

Results are provided for groups of students defined by shared characteristics—gender, race or ethnicity, parents' education level, and type of school. Based on participation rate criteria, results are reported for groups of students only when adequate school representation and sufficient numbers of students are present. The minimum requirement is at least 62 students in a particular student group from at least five primary sampling units (PSUs). However, the data for all students, regardless of whether their group was reported separately, were included in computing overall results. Explanations of the reporting groups are presented below.

Gender

Results are reported separately for males and females. Gender was reported by the student.

Race/Ethnicity

Results are presented for students in different racial/ethnic groups according to the following mutually exclusive categories: White, Black, and Hispanic. Results for Asian/Pacific Islander and American Indian (including Alaska Native) students are not reported separately because there were too few students in the groups for statistical reliability. The data for all students, regardless of whether their racial/ethnic group was reported separately, were included in computing the overall national results.

In NAEP long-term trend assessments, data about student race/ethnicity have been collected in three ways: through observation, school records, and student self-reports. The 2004 bridge assessment and all previous assessment results are based on observed race/ethnicity. The 2004 modified assessment results—shown in chapter 5 of the print report and used in the NAEP Questions Tool—are based on school-reported race/ethnicity. As in previous long-term trend assessments, data on students' self-reports of race/ethnicity were collected in 2004, but results are not reported based on self-reported racial/ethnic groups.

Parents' Education Level

Students were asked to indicate the extent of schooling for each of their parents—did not finish high school, graduated from high school, had some education after high school, or graduated from college. The response indicating the higher level of education was selected for reporting. Note that a substantial number of nine-year-olds indicated that they did not know their parents' education level; therefore, results are not presented for this age group.

Exclusion Rates and Assessment Results

Some students selected for participation in the NAEP long-term trend assessments were identified as English language learners (ELL) or students with disabilities (SD). In previous long-term trend assessments, if it was decided that a student classified as SD or ELL could not meaningfully participate in the NAEP assessment for which he or she was selected, the student was, according to NAEP guidelines, excluded from the assessment. See additional information about the percentage of students identified, excluded, and assessed in long-term trend mathematics and identified, excluded, and assessed in long-term trend reading.

For each student selected to participate in NAEP who was identified as either SD or ELL, a member of the school staff most knowledgeable about the student completed an SD/ELL questionnaire. Students with disabilities were excluded from the assessment if an IEP (individualized education program) team or equivalent group determined that the student could not participate in assessments such as NAEP; if the student's cognitive functioning was so severely impaired that he or she could not participate; or if the student's IEP required that the student be tested with an accommodation or adaptation not permitted or available in NAEP, and the student could not demonstrate his or her knowledge of the assessment subject area without that accommodation or adaptation.

A student who was identified as ELL and who was a native speaker of a language other than English was excluded if the student received instruction in the assessment subject area (e.g., reading or mathematics) primarily in English for less than three school years, including the current year, or if the student could not demonstrate his or her knowledge of reading or mathematics in English without an accommodation or adaptation.

In recent years, changes in policy and legislation pertaining to civil rights have resulted in assessment accommodations being permitted so that SD and ELL students selected to participate in NAEP can do so under modified conditions. Beginning with the 2004 modified trend assessments, NAEP long-term trend assessments offered accommodations for these students. For consistency with the long-term trend assessments administered in past years, however, accommodations were not offered to students in the 2004 bridge assessment samples.

Statistical Significance

The differences between scale scores and between percentages discussed in the results on this website take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics. Throughout the results, differences between scores or between percentages are discussed only when they are significant from a statistical perspective.

All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policymakers, educators, and the public.

Cautions in Interpretations

Users of this website are cautioned against interpreting NAEP results as implying causal relationships. Inferences related to student group performance or to the effectiveness of particular classroom practices, for example, should take into consideration the many socioeconomic and educational factors that may also impact performance.

The NAEP long-term trend scales make it possible to examine relationships between students' performance and various background factors measured by NAEP. However, a relationship that exists between achievement and another variable does not reveal its underlying cause, which may be influenced by a number of other variables. Similarly, the assessments do not reflect the influence of unmeasured variables. The results are most useful when they are considered in combination with other knowledge about the student population and the educational system, such as trends in instruction, changes in the school-age population, and societal demands and expectations.

Return to main results.