Increasing the Inclusion of Students with Disabilities and Limited English Proficient Students in NAEP

Vol. 2, No. 1, July 1996

The purpose of this edition of Focus on NAEP is to describe the activities underway in the National Assessment of Educational Progress (NAEP) toward increasing the numbers of students with disabilities (SD) or limited English proficient (LEP) students who are included in the assessment. In this article, information is provided on the changes made to the inclusion criteria, the types of accommodations now being offered, new procedures implemented in 1996, and ongoing research studies on inclusion issues.

NAEP is in the process of examining and implementing new procedures to maximize the representativeness of students included in the assessment. The focus is on efforts to enhance overall participation by including more SD and LEP students. Inclusion is receiving increased attention at the federal, state, and local levels, with much activity occurring in recent years. Changes have been made in the criteria for making selection decisions to include more students in the assessment and to offer various accommodations to students with special needs. Currently, NAEP is looking carefully into a variety of issues having to do with inclusion of both the SD and LEP student populations.

NAEP's main purpose is to provide key indicators of what the nation's students know and can do. This concept means that the NAEP results should represent all students in the nation. This is especially important because NAEP uses a sampling approach in which all students may be included, although, in actuality, some students with disabilities and limited English proficiency students do not participate in the assessment. Recent educational trends, reflected in the authorizations of Goals 2000: Educate America Act and Improving America's Schools Act (IASA), and the proposed reauthorization of the Individuals with Disabilities Education Act (IDEA), have called for assessments that are meaningful, challenging, and appropriate for all students. This call has led NAEP to look closely at the procedures for assessing larger numbers of SD and LEP students. NAEP is committed to increasing participation of as many of these students as possible in the assessment. Among the many benefits of this approach are improved measurement of overall student achievement, enhancement of the representativeness and generalizability of NAEP results, and greater fairness and equity. Among the challenges for NAEP are upholding its high degree of validity, maintaining its reliability, and preserving the ability to analyze and report trends in the face of changes made to procedures and the sampled population of respondents.

NAEP's Approach Prior to 1995

In the past, students with disabilities and limited English proficiency were often excluded from NAEP for several reasons--state and local policies had been designed to identify SD and LEP students and exclude them from testing based on certain criteria; school staff may have believed these students were unable to participate meaningfully; or, no test accommodations or adaptations were available that met the specific needs or requirements of the Individualized Education Plans (IEPs) required by law for students with certain disabilities. In order to standardize NAEP procedures, NAEP had previously developed policy guidelines for including students with disabilities or limited English proficiency, but some were excluded, particularly those with profound disabilities or students who would have required accommodations to the testing procedures.

In previous years, about half or more of the students identified as IEP or LEP were excluded from the NAEP assessments (see table 1). The percentages of students excluded, because in the judgment of school staff they could not participate meaningfully, have been fairly steady over time, with about five percent excluded due to IEPs and one percent excluded due to LEP. While the percentage of students excluded has remained relatively small in relation to the total population, these students make up a relatively large portion of the special education and LEP populations. Decisions to exclude students from the assessment were based on specific criteria used with NAEP (discussed further in the next section). Exclusion was further impacted by a lack of available accommodations and adaptations (such as extended testing time or assessment instruments in other languages).

Table 1.--NAEP Inclusion Rates for IEP and LEP Students, by Grade, 1992-1994

	Percent Students Identified IEP	Percent IEP Students Assessed	Percent Students Identified LEP	Percent LEP Students Assessed
Grade 4
1992	9	33	4	25
1994	12	50	6	50
Grade 8
1992	9	44	3	33
1994	13	38	4	50
Grade 12
1992	5	20	2	50
1994	11	36	3	66

An evaluation of NAEP conducted by the National Academy of Education¹ in 1994 found that many of the SD and LEP students who had been excluded from NAEP were, in fact, capable of participating in the assessment. This was particularly true if certain types of adaptations and accommodations could be offered for the assessment. It also recommended that NAEP develop better criteria to promote inclusion, rather than exclusion, of students. During this same time period, various offices in the Education Department as well as other groups also were involved in examining the issues related to including more students in assessments. These groups provided additional valuable input to NAEP on ways to improve the inclusion and accommodations process.² In response to these concerns, NAEP explored ways to increase the inclusion rates even further. To meet these objectives, plans were made to try out new approaches toward inclusiveness in preparation for the 1996 assessment.

Changes in Procedures in Preparation for the 1996 NAEP

The exclusion or inclusion criteria used in NAEP have changed somewhat over time. Prior to 1995, the procedures used by NAEP to determine who can participate in the assessment were based on criteria for excluding students. Beginning with the 1995 NAEP field test, the criteria were revised to be more inclusionary. Other changes were made to make a better link between subject areas of instruction and the assessment and in response to concerns about the language used in the old criteria. The old and new criteria are presented in table 2.

Table 2.--Old and New Exclusion or Inclusion Rules for NAEP

	Old (1990-1994)	New (1995-1996)
	EXCLUDE IF:	INCLUDE IF:
Students with Disabilities	Student is mainstreamed less than 50 percent of the time in academic subjects and is judged incapable of participating meaningfully in the assessment, OR IEP team or equivalent group determine that the student is incapable of participating meaningfully in the assessment.	Student has an IEP, unless the IEP team or equivalent group determine that the student cannot participate, or if the student's cognitive functioning is so severely impaired that he or she cannot participate, even with accommodations.
Students with Limited English Proficiency	Student is native speaker of a language other than English, AND Enrolled in an English-speaking school (not including bilingual education program) for less than two years, AND Judged to be incapable of taking part in the assessment.	Student has received academic instruction primarily in English for at least three years, OR Student has received academic instruction in English for less than three years, if school staff determine that the student is capable of participating in the assessment in English, OR Student, whose native language is Spanish, has received academic instruction in English for less than three years, if school staff determine that the student is capable of participating in the assessment in Spanish (if available).

In all years, schools were instructed that SD and LEP students should be assessed if, in the judgment of school staff, they were capable of taking the assessment, and that, in cases of doubt, the school should err on the side of inclusion.

In preparation for the 1996 assessment, NAEP field tested in 1995 the new inclusionary criteria for participation and the use of various accommodations and adaptations for the mathematics assessment. The field test also included science, but no accommodations or adaptations were tested for this subject. For mathematics, accommodations were made available for students with disabilities if they were part of the student's normal testing procedure, as specified in the student's IEP.

The accommodations for students with disabilities included

provision of large-print booklets and large-face calculators;
provision of Braille booklets and talking calculators;
accommodations in administration procedures, (e.g., unlimited testing time, individual or small-group administrations, allowing a facilitator to read directions, allowing students to give answers orally, allowing students to give answers using a special mechanical apparatus).

Accommodation and adaptation strategies for limited English proficient students (provided for mathematics only) included the availability of

Spanish-English bilingual assessment booklets, with items in different languages presented on facing pages;
Spanish-only assessment booklets.

In general, the 1995 field test results appear to be encouraging. Some SD and LEP students who would not have participated under previous assessment conditions were able to participate in the field test. The results of the field test showed that the new procedures could be implemented successfully in the 1996 national assessment. However, a preliminary analysis of the test items and student performance indicated that, for both the SD and LEP samples of students included and assessed under nonstandard conditions (i.e., with accommodations or adaptations) in the field test, the results may not be comparable to those from other students. Further study of the statistical and measurement issues was indicated for 1996.

Design, Inclusion Criteria, and Accommodations for the 1996 NAEP

In preparing for the 1996 assessment, NAEP staff were concerned about several conflicting issues related to possible changes that were being planned, including: (1) the measurement of trends, (2) the comparability of results from students who were accommodated with those who were not, and (3) the impact of changes in the inclusion criteria and provision of accommodations.

To address these issues, a special sample design was developed to examine carefully the effects of these changes. The 1996 NAEP was designed to allow measurement of trends from past years, necessitating following the same procedures as in the past, while at the same time incorporating new criteria for inclusion of students with disabilities and LEP students and offering accommodations and adaptations for students needing them. The design allowed research into the impact of changes in the inclusion criteria and the use of accommodations and adaptations. To address these needs, the 1996 national-level NAEP in mathematics actually consisted of three separate samples of about 3,500 students each at grades 4, 8, and 12, as shown in table 3:

Table 3.--Sample Design for 1996 NAEP Mathematics Assessment

Sample and Purpose	Inclusion Criteria	Accommodations
1 - Trends to past years	Old (1990-1994)	No
2 - Evaluate new criteria	New (1995-1996)	No
3 - Evaluate new criteria and accommodations	New (1995-1996)	Yes

In Sample 1, to measure trends to past years, SD and LEP students were included using the same criteria as in 1990-1994 and no accommodations were offered. In Sample 2, to evaluate the effect of using new inclusion criteria only, SD and LEP students were included using the new criteria, but no accommodations were offered. In Sample 3, to evaluate the impact of using new criteria and offering accommodations, SD and LEP students were included using the new criteria and accommodations were provided.

By comparing average results and inclusion rates among the three subsamples, the impact of the new inclusion criteria and of providing accommodations will be revealed. At the same time, results from students in the first sample and from similar students in the other two samples can be compared to results from previous years because testing procedures will not have changed for these students. In the 1996 science assessment, only the second and third subsamples were used because there were no previous assessments in science to which 1996 results will need to be compared.

Initial results pertaining to students with disabilities or limited English proficiency are planned to be reported in the 1996 NAEP Report Cards in mathematics and science in early 1997. One chapter in each report will describe procedures to increase inclusion and show the percentages of students in the subsamples who correctly answered selected assessment items. A later report will be devoted entirely to findings on the impact of including and accommodating SD and LEP students in the assessment.

Ongoing Research Studies on Inclusion Issues

Because this is a pioneering effort by NAEP, and examination of the topic is difficult because of its many complexities, additional research is required. A number of studies are currently underway in NAEP to investigate further issues related to inclusion of students with disabilities or limited English proficiency. This research includes studies of:

Scaling issues--an investigation to determine if there is a lack of fit between responses of students given under nonstandard assessment conditions with those given under standard conditions, and if so, what may account for it;
Reporting issues--an examination of the implications of reporting NAEP in alternate ways (e.g., percent-correct metric) for SD and LEP students when there is a lack of fit on the NAEP scale;
Appropriateness of the inclusion criteria for SD and LEP students--for students with disabilities, a determination of the proper role of the IEP team; for LEP students, a closer look at whether the threshold for English-language study should be two or three years;
Construct validity of the assessment for SD and LEP students--comparisons of the response patterns of students assessed under nonstandard conditions (e.g., with accommodations) with those of students assessed under standard conditions, and how they may differ; determination if NAEP is accurately measuring what accommodated students know and can do;
Language complexity issues--to examine whether native language proficiency of LEP students affects performance on NAEP and what language adaptations could be implemented into the assessment for these students; another study will examine if training scorers to recognize typical syntax, spelling, and other errors made by LEP students can improve the scoring of their responses to extended constructed-response items; and
Inclusion procedures--to investigate and determine if there still are students being excluded who could be included in the assessment.

It is hoped that this collection of studies will help NAEP make further progress toward the goal of full inclusion, while preserving the overall validity of the assessment. Research studies like these are essential in order to ensure that the reporting of what students know and can do, as well as gauging their academic progress over the years, is done in a reliable, valid, and meaningful way. As the various issues are examined and new procedures implemented, NAEP must maintain its commitment to high technical standards, the continuity of its trend data, and a balance of resources for the program.

In conclusion, the changes incorporated into the 1996 NAEP that further the goal of maximum inclusion of students with disabilities or limited English proficiency will result not only in an improved national assessment program, but will also benefit states, school districts, and other entities that conduct large-scale assessments. Many educators at these levels look to NAEP as a model for the best practice in assessment. Thus, NAEP needs to proceed in a thoughtful and thorough manner in its implementation of a more inclusive assessment.

Notes

1.The Trial State Assessment: Prospects and Realities. The Third Report of the National Academy of Education Panel on the Evaluation of the NAEP 1992 Trial State Assessment. (National Academy of Education, Stanford University: 1993).

2.(a) Making Decisions about the Inclusion of Students with Disabilities in Large-Scale Assessments--A Report on a Working Conference to Develop Guidelines on Inclusion and Accommodations. (National Center on Educational Outcomes, College of Education, University of Minnesota: 1994).

(b) Recommendations for Making Decisions about the Participation of Students with Disabilities in Statewide Assessment Programs--A Report on a Working Conference to Develop Guidelines for Statewide Assessments and Students with Disabilities. (National Center on Educational Outcomes, College of Education, University of Minnesota: 1994).

(c) Proceedings of the Conference on Inclusion Guidelines and Accommodations for Limited English Proficient Students in the National Assessment of Educational Progress. (National Center for Education Statistics, OERI, U.S. Dept. of Education: 1996).

The Focus on NAEP series briefly summarizes information about the ongoing development and implementation of the National Assessment of Educational Progress. The series is a product of the National Center for Education Statistics. Gary W. Phillips serves as the Associate Commissioner for Education Assessment. This issue was written by John F. Olson of the Education Statistics Services Institute (ESSI) and Arnold A. Goldstein of the National Center for Education Statistics.

For questions, and for more information about NAEP and to order NAEP products see the NAEP staff directory