Skip Navigation

Illustration/Logo View Quarterly by  This Issue  |  Volume and Issue  |  Topics
Education Statistics Quarterly
Vol 5, Issue 1, Topic: Elementary and Secondary Education
Including Special-Needs Students in the NAEP 1998 Reading Assessment, Part I, Comparison of Overall Results With and Without Accommodations
By: Anthony D. Lutkus and John Mazzeo
 
This article was originally published as the Executive Summary of the Statistical Analysis Report of the same name. The sample survey data are from the National Assessment of Educational Progress (NAEP) 1994 and 1998 Reading Assessments.
 
 

The National Assessment of Educational Progress (NAEP) is the nation’s only ongoing survey of student achievement in core subject areas. Authorized by Congress, administered by the National Center for Education Statistics (NCES) in the U.S. Department of Education, and overseen by the National Assessment Governing Board (NAGB), NAEP regularly reports to the public on the educational progress of a representative sample of students in grades 4, 8, and 12.

Because NAEP’s purpose is to report on what students know and can do, it is important that its student samples and assessment results represent the performance of all students. This includes the results for special-needs students—students with disabilities (SD) and limited-English-proficient (LEP) students. Although the intent of NAEP has consistently been to include special-needs students in its assessments to the fullest degree possible, the implementation of assessments has resulted in some exclusion of SD and LEP students. In order to participate in the NAEP assessments, some special-needs students require accommodations in the test administration. In 1996, NAEP began offering accommodations on a trial basis and conducting research to explore possible psychometric effects that the inclusion of accommodated special-needs students might have on assessment results in various subject areas.

The NAEP 1998 Reading Report Card for the Nation and the States (Donahue et al.1999) included national results for fourth-, eighth-, and twelfth-graders, as well as results for fourth- and eighth-graders in those states and other jurisdictions (i.e., U.S. territories and the District of Columbia) that volunteered to participate in state-level assessments.* In order to allow comparisons with results in 1992 and 1994, when accommodations were not offered, the report card did not incorporate the 1998 results for special-needs students who were tested with accommodations.

back to top


There are two purposes to this report. The first is to present NAEP 1998 Reading Assessment results that are recalculated to include results from special-needs students who were tested with accommodations. The second is to examine the impact on NAEP results of the varying exclusion rates of special-needs students, by participating states and other jurisdictions. A follow-up report will explore the patterns of results for accommodated versus nonaccommodated samples separately for students who are SD and LEP, using combined national and state data.


Performance results for both the national and state assessments are reported in two ways. The first is in terms of average scale scores on the NAEP reading composite scale, which ranges from 0 to 500. The second is in terms of percentages of students at or above each of NAGB’s three achievement levels (Basic, Proficient, and Advanced). The following is a summary of the findings addressed in this report:

  • There were no statistically significant differences between the originally reported national average scale scores (where accommodations were not permitted) and the recalculated average scores (including data from the administrations where accommodations were permitted) at any of the three grades (table A). Further, there were no statistically significant differences in the percentage of students at or above the Basic or Proficient achievement levels for the two samples at any of the grades assessed.
  • In contrast to the unchanged results for the national data, at grade 4, average scale scores were higher in nine states for the original samples without accommodations permitted than for the samples with accommodations permitted (table B). At grade 8, there were no statistically significant differences in average scale scores between the two sample types in any state or jurisdiction. At both grades 4 and 8, there were no statistically significant differences in the percentage of students at or above the Basic or Proficient achievement levels in any state or jurisdiction.
  • There were no statistically significant differences in national average reading scale scores between the two sample types at any grade for either male or female students.
  • At the state level, however, average reading scores at grade 4 were higher for male students in three states when accommodations were not permitted, higher for female students in four states when accommodations were not permitted, and higher for both male and female students in one state when accommodations were not permitted. No statistically significant differences in the sample types by gender were found in grade 8.
  • No statistically significant differences were found in the national data between the two sample types for any ethnic group in any grade.
  • The state-level results showed statistically significant differences by race between the two sample types in five states at grade 4. In four states, Black students in the accommodated samples had a lower average scale score than their peers in the samples where accommodations were not permitted. In three states, White students in the accommodated samples had a lower average scale score than their peers in the samples where accommodations were not permitted. In two of the five states noted above, both Black students and White students at grade 4 had lower average scale scores when accommodations were permitted. At grade 8, there were no statistically significant differences between the two sample types by ethnic group in any state or jurisdiction.
  • In three states, fourth-graders who were eligible for free/reduced-price lunch under the National School Lunch Program had higher average reading scores when accommodations were not permitted than when accommodations were permitted. At grade 8, there were no statistically significant differences in the average scores of students in the two sample types by eligibility for free/reduced-price school lunch.

Table A. National average reading scores, and exclusion and accommodation rates, when accommodations were not permitted and when accommodations were permitted, grades 4, 8, and 12: 1998
  Average scale scores Exclusion rates* Accommodation
rates*
Accommodations
not permitted
Accommodations
permitted
Accommodations
not permitted
Accommodations
permitted
Grade 4
217 216 9 6 3
Grade 8
264 263 6 4 2
Grade 12
291 290 3 2 1

*Combined rate for SD and LEP students.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. (Originally published as table 2.1 on p. 16 of the complete report from which this article is excerpted.)


Table B. Average reading scores by state, when accommodations were not permitted and when accommodations were permitted, grade 4: 1998
  Accommodations not permitted Accommodations permitted
Number of
students
Average
score
Number of
students
Average
score
Alabama
2,506 211 2,475 211
Arizona
2,432 207 2,423 206
Arkansas
2,580 209 2,573 209
California
1,722 202 1,713 202
Colorado
2,528 222 2,540 220
Connecticut
2,484 232 2,531 230
Delaware
2,309 212 2,359 207a
Florida
2,463 207 2,532 206
Georgia
2,647 210 2,684 209
Hawaii
2,600 200 2,601 200
Iowa
2,232 223 2,244 220a
Kansas
1,845 222 1,856 221
Kentucky
2,442 218 2,456 218
Louisiana
2,587 204 2,658 200A
Maine
2,355 225 2,386 225
Maryland
2,241 215 2,308 212a
Massachusetts
2,306 225 2,325 223a
Michigan
2,365 217 2,368 216
Minnesota
2,271 222 2,292 219a
Mississippi
2,552 204 2,554 203
Missouri
2,482 216 2,514 216
Montana
1,847 226 1,886 225
Nevada
2,597 208 2,613 206
New Hampshire
1,805 226 1,805 226
New Mexico
2,284 206 2,333 205
New York
2,221 216 2,256 215
North Carolina
2,514 217 2,552 213a
Oklahoma
2,576 220 2,553 219
Oregon
2,396 214 2,351 212a
Rhode Island
2,533 218 2,500 218
South Carolina
2,411 210 2,433 209
Tennessee
2,627 212 2,599 212
Texas
2,241 217 2,267 214
Utah
2,678 215 2,642 216
Virginia
2,602 218 2,625 217
Washington
2,378 217 2,340 218
West Virginia
2,518 216 2,531 216
Wisconsin
2,071 224 2,126 222a
Wyoming
2,642 219 2,635 218
Other jurisdictions
District of Columbia
2,353 182 2,397 179
DDESS
2,647 220 2,628 219
DoDDS
2,609 223 2,623 221
Virgin Islands
1,469 178 1,468 174

Indicates jurisdiction did not meet one or more of the guidelines for school participation. See the NAEP 1998 Reading Report Card for the Nation and the States (Donahue et al. 1999, p.155).

A/a = Significantly different from the original sample.
A = Significance level is adjusted for multiple comparisons across jurisdictions.
a = Pairwise significance test not adjusted for multiple comparisons.

DDESS: Department of Defense Domestic Dependent Elementary and Secondary Schools.

DoDDS: Department of Defense Dependents Schools (Overseas).

NOTE: Differences between states and jurisdictions may be partially explained by other factors not included in this table.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. (Originally published as table 2.2 on p. 17 of the complete report from which this article is excerpted.)

back to top


Evolving policies and practices regarding the inclusion of special-needs students pose challenges to the state NAEP program as it strives to monitor accurately trends in academic achievement. As policies and practices have changed, the state NAEP program has seen corresponding changes in the proportion of special-needs students included in its samples. This report provides data on statistically significant differences in exclusion rates when accommodations were not permitted and when accommodations were permitted, and the relationship between those exclusion rates and state average scale scores.

  • Allowing accommodations in the reading assessment resulted in decreased exclusion rates for a number of participating states and other jurisdictions (table C). At grade 4, 12 of the 43 states or other jurisdictions that participated in the 1998 reading assessment had exclusion percentages of 10 percent or higher when accommodations were not permitted. In contrast, only five jurisdictions had exclusion percentages as high when accommodations were permitted. Differences in exclusion rates by accommodation were statistically significant in 10 of 43 jurisdictions at grade 4 and 10 of 40 jurisdictions at grade 8.
  • While no state or other jurisdiction tested large percentages of students with accommodations, considerable variability was evident. At grade 4, two jurisdictions tested more than 5 percent of students with accommodations, while two jurisdictions tested 1 percent or less. At grade 8, there was generally less use of accommodations, although 7 of 40 participating jurisdictions tested 4 percent or more of students with accommodations.
  • There is a negative relationship between the change in inclusion rates and difference in average scale scores. States or other jurisdictions that exhibited larger gains in inclusion (i.e., lower exclusion rates) tended to also exhibit larger reduction in their average scale scores when accommodations were permitted. The correlation was -0.72 at fourth grade and
    -0.52 at eighth grade.

Table C. Reading exclusion rates by state for special-needs students, when accommodations were not permitted and when accommodations were permitted, grades 4 and 8: 1998

  Grade 4 Grade 8
Accommodations
not permitted
Accommodations
permitted
Accommodations
not permitted
Accommodations
permitted
National public and
nonpublic
9 6 6 4
National public
10 7 6 4
Alabama
8 8 6 6
Arizona
10 10 7 5
Arkansas
5 5 7 5
California
15 14 8 4
Colorado
7 6 5 4
Connecticut
13 10 8 6
Delaware
7 1 6 2
Florida
9 6 5 5
Georgia
7 5 5 4
Hawaii
5 5 6 5
Iowa
8 5 * *
Kansas
6 4 5 4
Kentucky
9 7 5 3
Louisiana
12 7 10 5
Maine
8 7 7 5
Maryland
10 6 7 3
Massachusetts
8 5 7 4
Michigan
7 6 * *
Minnesota
4 3 4 1
Mississippi
4 4 7 6
Missouri
7 6 6 4
Montana
4 2 3 4
Nevada
12 11 8 6
New Hampshire
5 3 * *
New Mexico
11 9 7 8
New York
9 7 10 8
North Carolina
10 7 9 6
Oklahoma
9 9 9 9
Oregon
7 6 4 4
Rhode Island
7 7 5 6
South Carolina
11 8 6 5
Tennessee
4 4 4 6
Texas
14 13 7 5
Utah
5 6 5 4
Virginia
8 6 7 5
Washington
5 5 4 4
West Virginia
9 8 8 7
Wisconsin
10 8 8 5
Wyoming
4 3 2 2
Other jurisdictions
District of Columbia
11 9 9 5
DDESS
5 4 5 2
DoDDS
4 3 4 1
Virgin Islands
6 5 7 7

*State did not participate at grade 8.

DDESS: Department of Defense Domestic Dependent Elementary and Secondary Schools.

DoDDS: Department of Defense Dependents Schools (Overseas).

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. (Originally published as table 2.4 on p. 21 of the complete report from which this article is excerpted.)

back to top


Offering accommodations in state NAEP to students who receive them in their regular classroom assessments increases inclusion in some states and other jurisdictions, but the magnitude of the increase varies across jurisdictions. At grade 4, the increase in inclusion of special-needs students and the provision of accommodations was associated with lower average scale scores in nine states, but not in the nation. At grade 8, there was no pattern of statistically significant differences by accommodation status.

Readers are cautioned not to overgeneralize the results of this study for several reasons. First, within the reading subject area, the lack of difference between samples with accommodations permitted and those without accommodations permitted in the national data in 1998 did not continue in the 2000 fourth-grade national assessment, where a statistically significant difference did appear in favor of the sample without accommodations. Second, patterns may vary depending on the academic subject area. For example, in the present discussion of 1998 state reading assessment results, more states at grade 4 had lower average scale scores for their accommodated samples than at grade 8. However, for the mathematics assessment in 2000, the findings were different—more states at grade 8 had statistically significant lower average scale scores in accommodated samples than at grade 4. Finally, the fact that the reading assessment did not permit a Spanish translation suggests that the findings for LEP students may not generalize to other subjects, such as mathematics, where this accommodation may be offered.

back to top


Footnote

* In this report, the term jurisdiction is sometimes used to refer to both states and other jurisdictions.


Donahue, P.L., Voelkl, K.E., Campbell, J.R., and Mazzeo, J. (1999). NAEP 1998 Reading Report Card for the Nation and the States (NCES 1999–500). U.S. Department of Education. Washington, DC: National Center for Education Statistics.

back to top

Data source: The National Assessment of Educational Progress (NAEP) 1994 and 1998 Reading Assessments.

For technical information, see the complete report:

Lutkus, A.D., and Mazzeo, J. (2003). Including Special-Needs Students in the NAEP 1998 Reading Assessment: Part I, Comparison of Overall Results With and Without Accommodations (NCES 2003–467).

Author affiliations: A.D. Lutkus and J. Mazzeo, Educational Testing Service.

For questions about content, contact Arnold Goldstein (arnold.goldstein@ed.gov).

To obtain the complete report (NCES 2003–467), call the toll-free ED Pubs number (877–433–7827) or visit the NCES Electronic Catalog (http://nces.ed.gov/pubsearch).


back to top