Table of Contents  |  Search Technical Documentation  |  References

NAEP Sample Design → Sample Design for the 2000 Assessment → National Main Assessment Sample Design in 2000 → Sampling Frame for the 2000 National Main Assessment → School Substitution for Adjusting School Nonresponse in the 2000 National Main Assessment

NAEP Technical DocumentationSchool Substitution for Adjusting School Nonresponse in the 2000 National Main Assessment

			Original Sample Schools With and Without Substitutes

NAEP assessments traditionally use substitute schools to compensate for school-level nonresponse and to improve the overall yield of assessed students. A substitute school is a school that can replace an original school if that original school refuses to participate in the assessment. Therefore, a nonresponding original sample school is replaced with another school that can be anticipated as yielding similar assessment results. If this is really the case, then using the substitute will reduce nonresponse bias.

Potential substitute schools were selected for all sampled schools in the national main assessment where a close match could be identified. A new procedure was introduced for the 2000 assessment to identify substitutes. No sampled school was assigned more than one substitute, and no school was assigned to be a substitute for more than one school. The criteria for assigning substitutes were quite strict. Many sampled schools were not assigned substitutes at all, as there were no schools that met the necessary criteria to be a substitute.

The potential pool of substitute schools for the national main assessment for a given grade was restricted to schools with that grade that were not part of any of the NAEP samples (state or main) for any grade. The substitutes were drawn in a single step from a joint pool of unsampled schools, with twelfth-grade sampled schools receiving substitutes first, eighth-grade sampled schools next, and fourth-grade schools last.

There were a number of other "absolute" boundaries, as follows:

• Only schools within the same school type group could serve as substitutes. The school type groups in the 2000 national main assessment comprised

• regular public schools,

• Bureau of Indian Affairs schools,

• Department of Defense Education Activity schools,

• other public schools,

• Catholic schools,

• non-Catholic religious schools, and

• other private schools.

• Only those public schools that

  • were located in the same primary sampling unit (PSU),

  • had the same locality type, and

  • had the same minority status

• as the originally sampled school(s) could serve as substitute schools.

• Catholic schools could only have as substitutes schools in the same district (usually diocese).

Within the various cells defined by these absolute boundaries, a distance measure was computed for each sampled school and its potential matches. The distance measure was computed based on percent Blacks, percent Hispanics, and the square root of the grade enrollment, and used a standardized Euclidean distance D_ij as follows:

• D_ij is the distance between the i^th sampled school and the j^th candidate substitute;

• B_i is the percentage of Black students in sampled school i, H_i the percentage of Hispanic students in sampled school i, E_i the school grade enrollment in sampled school i;

• B_j is the percentage of Black students in candidate substitute school j, H_j the percentage of Hispanic students in candidate substitute school j, E_j the school grade enrollment in candidate substitute school j;

• VBH is the average of the mean variance of Black percentages across all schools within the grade and the mean variance of Hispanic percentages across all schools within the grade; and

• VSQE is the mean variance of the square root of estimated grade enrollment across all schools within the grade.

Two passes were carried out for each grade. The first pass was a matching of regular public schools to schools outside their district. From a field viewpoint, it is preferable to have substitutes in different districts as many public school districts refuse to allow substitution. NAEP took an out-of-district substitute j for a sampled public school i if the pair had a D_ij value less than 0.65. If there was more than one candidate for a particular sampled school i, then the j-school with the smallest D_ij was selected. (Note that a particular j-school might be the best substitute for two or more sampled schools. In this case, the school was matched with the sampled school in which it had the smallest D_ij value. Because of this, some sampled schools did not receive as substitutes their best matches, as those schools might have already been assigned to other sampled schools.)

All matched schools in the first pass were removed from further consideration. The second pass was then a matching of regular public schools within their districts and of all private schools (Catholic schools within their dioceses, religious non-Catholic schools in one cell, and nonreligious private schools in one cell). The D_ij limit was 0.65: if a sampled school had no potential substitute lower than this limit, then that sampled school received no substitute. (Note that a school might have a j-school with a D_ij value less than 1.00 that is "lost" to another school and may thereby have no substitute.)

Although the selected cutoff point of 0.65 on the distance measure is somewhat arbitrary, it was decided upon based on experience with this algorithm in the 1994 Trial State Assessment (Mazzeo, Allen, and Kline 1995) by a group of statisticians finding consensus on the distance measures at which substitutes began to appear unacceptable.

Printer-friendly Version