The purpose of this edition of Focus on NAEP is to describe the activities underway in the National Assessment of Educational Progress (NAEP) toward increasing the numbers of students with disabilities (SD) or limited English proficient (LEP) students who are included in the assessment. In this article, information is provided on the changes made to the inclusion criteria, the types of accommodations now being offered, new procedures implemented in 1996, and ongoing research studies on inclusion issues.
NAEP is in the process of examining and implementing new procedures to maximize the representativeness of students included in the assessment. The focus is on efforts to enhance overall participation by including more SD and LEP students. Inclusion is receiving increased attention at the federal, state, and local levels, with much activity occurring in recent years. Changes have been made in the criteria for making selection decisions to include more students in the assessment and to offer various accommodations to students with special needs. Currently, NAEP is looking carefully into a variety of issues having to do with inclusion of both the SD and LEP student populations.
NAEP's main purpose is to provide key indicators of what the nation's
students know and can do. This concept means that the NAEP results should
represent all students in the nation. This is especially important
because NAEP uses a sampling approach in which all students may be included,
although, in actuality, some students with disabilities and limited English
proficiency students do not participate in the assessment. Recent educational
trends, reflected in the authorizations of Goals 2000: Educate America Act and
Improving America's Schools Act (IASA), and the proposed reauthorization of the
Individuals with Disabilities Education Act (IDEA), have called for assessments that are
meaningful, challenging, and appropriate for all students. This call has
led NAEP to look closely at the procedures for assessing larger numbers of SD
and LEP students. NAEP is committed to increasing participation of as many of
these students as possible in the assessment. Among the many benefits of this
approach are improved measurement of overall student achievement, enhancement of
the representativeness and generalizability of NAEP results, and greater
fairness and equity. Among the challenges for NAEP are upholding its high degree
of validity, maintaining its reliability, and preserving the ability to analyze
and report trends in the face of changes made to procedures and the sampled
population of respondents.
In the past, students with disabilities and limited English proficiency were often excluded from NAEP for several reasons--state and local policies had been designed to identify SD and LEP students and exclude them from testing based on certain criteria; school staff may have believed these students were unable to participate meaningfully; or, no test accommodations or adaptations were available that met the specific needs or requirements of the Individualized Education Plans (IEPs) required by law for students with certain disabilities. In order to standardize NAEP procedures, NAEP had previously developed policy guidelines for including students with disabilities or limited English proficiency, but some were excluded, particularly those with profound disabilities or students who would have required accommodations to the testing procedures.
In previous years, about half or more of the students identified as IEP or LEP were excluded from the NAEP assessments (see table 1). The percentages of students excluded, because in the judgment of school staff they could not participate meaningfully, have been fairly steady over time, with about five percent excluded due to IEPs and one percent excluded due to LEP. While the percentage of students excluded has remained relatively small in relation to the total population, these students make up a relatively large portion of the special education and LEP populations. Decisions to exclude students from the assessment were based on specific criteria used with NAEP (discussed further in the next section). Exclusion was further impacted by a lack of available accommodations and adaptations (such as extended testing time or assessment instruments in other languages).
Table 1.--NAEP Inclusion Rates for IEP and LEP Students, by Grade, 1992-1994
||Percent Students Identified IEP||Percent IEP Students Assessed||Percent Students Identified LEP||Percent LEP Students Assessed|
An evaluation of NAEP conducted by the National Academy of Education1 in 1994 found that many of the SD and LEP students who had been excluded from NAEP were, in fact, capable of participating in the assessment. This was particularly true if certain types of adaptations and accommodations could be offered for the assessment. It also recommended that NAEP develop better criteria to promote inclusion, rather than exclusion, of students. During this same time period, various offices in the Education Department as well as other groups also were involved in examining the issues related to including more students in assessments. These groups provided additional valuable input to NAEP on ways to improve the inclusion and accommodations process.2 In response to these concerns, NAEP explored ways to increase the inclusion rates even further. To meet these objectives, plans were made to try out new approaches toward inclusiveness in preparation for the 1996 assessment.
The exclusion or inclusion criteria used in NAEP have changed somewhat over time. Prior to 1995, the procedures used by NAEP to determine who can participate in the assessment were based on criteria for excluding students. Beginning with the 1995 NAEP field test, the criteria were revised to be more inclusionary. Other changes were made to make a better link between subject areas of instruction and the assessment and in response to concerns about the language used in the old criteria. The old and new criteria are presented in table 2.
Table 2.--Old and New Exclusion or Inclusion Rules for NAEP
||Old (1990-1994)||New (1995-1996)|
|Students with Disabilities||
Student is mainstreamed less than 50 percent of the time in academic subjects
and is judged incapable of participating meaningfully in the assessment, OR
IEP team or equivalent group determine that the student is incapable of participating meaningfully in the assessment.
|Student has an IEP, unless the IEP team or equivalent group determine that the student cannot participate, or if the student's cognitive functioning is so severely impaired that he or she cannot participate, even with accommodations.|
|Students with Limited English Proficiency||
Student is native speaker of a language other than English, AND
Enrolled in an English-speaking school (not including bilingual education program) for less than two years, AND
Judged to be incapable of taking part in the assessment.
Student has received academic instruction primarily in English for at least
three years, OR Student has received academic instruction in English for less than three
years, if school staff determine that the student is capable of participating in
the assessment in English, OR
Student, whose native language is Spanish, has received academic instruction in English for less than three years, if school staff determine that the student is capable of participating in the assessment in Spanish (if available).
In all years, schools were instructed that SD and LEP students should be assessed if, in the judgment of school staff, they were capable of taking the assessment, and that, in cases of doubt, the school should err on the side of inclusion.
In preparation for the 1996 assessment, NAEP field tested in 1995 the new inclusionary criteria for participation and the use of various accommodations and adaptations for the mathematics assessment. The field test also included science, but no accommodations or adaptations were tested for this subject. For mathematics, accommodations were made available for students with disabilities if they were part of the student's normal testing procedure, as specified in the student's IEP.
The accommodations for students with disabilities included
Accommodation and adaptation strategies for limited English proficient students (provided for mathematics only) included the availability of
In general, the 1995 field test results appear to be encouraging. Some SD and LEP students who would not have participated under previous assessment conditions were able to participate in the field test. The results of the field test showed that the new procedures could be implemented successfully in the 1996 national assessment. However, a preliminary analysis of the test items and student performance indicated that, for both the SD and LEP samples of students included and assessed under nonstandard conditions (i.e., with accommodations or adaptations) in the field test, the results may not be comparable to those from other students. Further study of the statistical and measurement issues was indicated for 1996.
In preparing for the 1996 assessment, NAEP staff were concerned about several conflicting issues related to possible changes that were being planned, including: (1) the measurement of trends, (2) the comparability of results from students who were accommodated with those who were not, and (3) the impact of changes in the inclusion criteria and provision of accommodations.
To address these issues, a special sample design was developed to examine carefully the effects of these changes. The 1996 NAEP was designed to allow measurement of trends from past years, necessitating following the same procedures as in the past, while at the same time incorporating new criteria for inclusion of students with disabilities and LEP students and offering accommodations and adaptations for students needing them. The design allowed research into the impact of changes in the inclusion criteria and the use of accommodations and adaptations. To address these needs, the 1996 national-level NAEP in mathematics actually consisted of three separate samples of about 3,500 students each at grades 4, 8, and 12, as shown in table 3:
Table 3.--Sample Design for 1996 NAEP Mathematics Assessment
|Sample and Purpose||Inclusion Criteria||
|1 - Trends to past years||Old (1990-1994)||No|
|2 - Evaluate new criteria||New (1995-1996)||No|
|3 - Evaluate new criteria and accommodations||New (1995-1996)||Yes|
In Sample 1, to measure trends to past years, SD and LEP students were included using the same criteria as in 1990-1994 and no accommodations were offered. In Sample 2, to evaluate the effect of using new inclusion criteria only, SD and LEP students were included using the new criteria, but no accommodations were offered. In Sample 3, to evaluate the impact of using new criteria and offering accommodations, SD and LEP students were included using the new criteria and accommodations were provided.
By comparing average results and inclusion rates among the three subsamples, the impact of the new inclusion criteria and of providing accommodations will be revealed. At the same time, results from students in the first sample and from similar students in the other two samples can be compared to results from previous years because testing procedures will not have changed for these students. In the 1996 science assessment, only the second and third subsamples were used because there were no previous assessments in science to which 1996 results will need to be compared.
Initial results pertaining to students with disabilities or limited English proficiency are planned to be reported in the 1996 NAEP Report Cards in mathematics and science in early 1997. One chapter in each report will describe procedures to increase inclusion and show the percentages of students in the subsamples who correctly answered selected assessment items. A later report will be devoted entirely to findings on the impact of including and accommodating SD and LEP students in the assessment.
Because this is a pioneering effort by NAEP, and examination of the topic is difficult because of its many complexities, additional research is required. A number of studies are currently underway in NAEP to investigate further issues related to inclusion of students with disabilities or limited English proficiency. This research includes studies of:
It is hoped that this collection of studies will help NAEP make further progress toward the goal of full inclusion, while preserving the overall validity of the assessment. Research studies like these are essential in order to ensure that the reporting of what students know and can do, as well as gauging their academic progress over the years, is done in a reliable, valid, and meaningful way. As the various issues are examined and new procedures implemented, NAEP must maintain its commitment to high technical standards, the continuity of its trend data, and a balance of resources for the program.
In conclusion, the changes incorporated into the 1996 NAEP that further the
goal of maximum inclusion of students with disabilities or limited English
proficiency will result not only in an improved national assessment program, but
will also benefit states, school districts, and other entities that conduct
large-scale assessments. Many educators at these levels look to NAEP as a model
for the best practice in assessment. Thus, NAEP needs to proceed in a thoughtful
and thorough manner in its implementation of a more inclusive assessment.
1.The Trial State Assessment: Prospects and Realities. The Third Report
of the National Academy of Education Panel on the Evaluation of the NAEP 1992
Trial State Assessment. (National Academy of Education, Stanford University:
2.(a) Making Decisions about the Inclusion of Students with Disabilities
in Large-Scale Assessments--A Report on a Working Conference to Develop
Guidelines on Inclusion and Accommodations. (National Center on Educational
Outcomes, College of Education, University of Minnesota: 1994).
(b) Recommendations for Making Decisions about the Participation of
Students with Disabilities in Statewide Assessment Programs--A Report on a
Working Conference to Develop Guidelines for Statewide Assessments and Students
with Disabilities. (National Center on Educational Outcomes, College of
Education, University of Minnesota: 1994).
(c) Proceedings of the Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in the National Assessment of Educational Progress. (National Center for Education Statistics, OERI, U.S. Dept. of Education: 1996).
The Focus on NAEP series briefly summarizes information about the ongoing development and implementation of the National Assessment of Educational Progress. The series is a product of the National Center for Education Statistics. Gary W. Phillips serves as the Associate Commissioner for Education Assessment. This issue was written by John F. Olson of the Education Statistics Services Institute (ESSI) and Arnold A. Goldstein of the National Center for Education Statistics.
For questions, and for more information about NAEP and to order NAEP products see the NAEP staff directory