Table of Contents  |  Search Technical Documentation  |  References

NAEP Sample Design → Sample Design for the 2000 Assessment → State Assessment Sample Design in 2000 → School Sample Selection for the 2000 State Assessment → School Substitution and Retrofitting in the 2000 State Assessment

NAEP Technical DocumentationSchool Substitution and Retrofitting in the 2000 State Assessment

			Number of Substitute Schools by Jurisdiction

Substitution for sampled schools that do not participate in the assessment provides a means to reduce the nonresponse bias that may result from school nonparticipation, and to maintain an acceptable yield of assessed students. Substitution cannot eliminate school nonresponse bias. However, to the extent that a nonparticipating school can be replaced in the sample by a school, not originally selected in the sample, that has similar characteristics, nonresponse bias is likely to be reduced. Thus the 2000 NAEP state assessment included a procedure for associating a potential substitute school with each school in the sample, where feasible.

An automated substitute selection mechanism assigns each sampled school a substitute school, if possible, before the field period begins. Schools and their substitutes share the same jurisdiction and urbanization classification. If possible, substitutes for public schools come from public schools in other school districts. This different district strategy arises from the fact that public school nonresponse may be due to district-level refusal.

A distance measure between a sampled school and each potential substitute school determines how well the schools match each other. This measure incorporates differences between the paired schools among four component variables:

• estimated grade enrollment,

• achievement data or median household income,

• largest student minority enrollment, and

• second largest student minority enrollment.

Each difference was squared and standardized to the population standard deviation for the component variable across all grade-eligible schools and jurisdictions. The distance measure equals the sum of the four resulting squared standardized differences.

For example, compare the following two schools from the NAEP 2000 state assessment:

School A:

• Estimated grade enrollment = 49

• Achievement score = 221

• Largest minority enrollment percentage = 29

• Second largest minority enrollment percentage = 24

School B:

• Estimated grade enrollment = 49

• Achievement score = 223

• Largest minority enrollment percentage = 25

• Second largest minority enrollment percentage = 21

Using the population standard deviation calculated for each component variable in the NAEP 2000 assessment, the distance measure between schools A and B equals 0.40. School B would be assigned as a substitute school for school A if no other school had a smaller distance measure.

The automated substitute selection mechanism chooses the potential substitute with the smallest distance measure as the substitute for a given school. The mechanism assigns only one substitute school to a sampled school. Once a school has been assigned as a substitute, it can not be used again. For the first pass, the maximum acceptable distance is .60, and public school substitutes must come from a different school district than the sampled school. Some sampled schools do not receive an assigned substitute school in the first pass. Either the sampled schools outnumber the substitute schools, or the distance measure exceeds .60 for the remaining substitute schools.

For the second pass, the mechanism raises the maximum acceptable distance measure to .75. It also lifts the different district constraint for public schools. These steps produce a small number of additional assigned substitutes.

Note the selected distance measure cutoff points of .60 and .75 started with the 1994 trial state assessment. Before that assessment, statisticians reviewed a large number of school listings, calculated the distance measures between school pairs, and agreed on the cutoff points at which substitute schools appear unacceptable. The selected cutoff points have been used since the 1994 assessment. More information can be found in Technical Report of the NAEP 1994 Trial State Assessment Program in Reading (Mazzeo, Allen, and Kline 1995).

When substitutes were assigned, it was sometimes the case that there was just a single school that was a suitable substitute for two sampled schools. Thus the sampled school that had the smaller distance measure to that potential substitute (school A) received it as an assigned substitute, while the second sampled school (school B) received no substitute. Once the field period began, on occasion it transpired that school A participated, while school B did not. In a case such as this, the substitute school was reassigned, and became a substitute for school B. In this way substitute schools were assigned to as many sampled schools as possible.

Printer-friendly Version