I. TIMSS POPULATION 2: NATIONS WITH DEVIATIONS FROM INTERNATIONAL STUDY GUIDELINES

Twenty-two of the 41 TIMSS countries experienced a more or less serious deviation from international guidelines for execution of the study at the Population 2 level (the two grades with the largest proportion of 13-year-olds—grades 7 and 8 in most countries). In 16 countries, the TIMSS International Study Center considered the deviations to be sufficiently serious to raise questions about the confidence to be placed in their Population 2 scores. These 16 nations with major difficulties are noted with an asterisk in this appendix.

*Australia—Participation rate did not meet the international criterion of 75 percent of schools and students combined. Participation rate was 70 percent after replacements for refusals were substituted.

*Austria—Participation rate did not meet the international criterion of at least 50 percent participation by schools before replacement. The initial school participation rate was 41 percent before replacement.

Belgium (Flemish)—Participation rate of 75 percent of schools and students combined was achieved only after replacements for refusals were substituted.

*Belgium (French)—Participation rate did not meet the international criterion of 75 percent of schools and students combined. Participation rate was 72 percent after replacements for refusals were substituted.

*Bulgaria—Participation rate did not meet the international criterion of 75 percent of schools and students combined. Participation rate was 63 percent after replacements for refusals were substituted.

*Colombia—The pair of grades tested was one grade higher than the international target. Average age of students in the upper grade was 15.7.

*Denmark—International guidelines requiring random selection of the classrooms to receive the assessment were not followed.

England—More than the international criterion of 10 percent of schools and students were excused from the test for various reasons, with resulting coverage of 89 percent of the desired population. Participation rate of 75 percent of schools and students combined was achieved only after replacements for refusals were substituted.

*Germany—The pair of grades tested was one grade higher than the international target. Average student age of students in the upper grade was 14.8. One of 16 regions (Baden-Wuerttemberg) did not participate in the study, with resulting coverage of 88 percent of the desired population. Participation rate of 75 percent of schools and students combined was achieved only after replacements for refusals were substituted.

*Greece—International guidelines requiring random selection of the classrooms to receive the assessment were not followed.

*Israel—Test administered only in the Hebrew-speaking public school system, with resulting coverage of 74 percent of the desired population. International guidelines requiring random selection of the classrooms to receive the assessment were not followed. Participation rate did not meet the international criteria of at least 50 percent participation by schools in the sample before replacement. The school participation rate before replacement was 45 percent.

*Kuwait—In contrast to other nations, which tested two adjacent grades, Kuwait tested only one grade, the ninth grade. This grade was higher than either of the grades that should have been the international target. Average student age was 15.3.

Latvia—LSS—Test administered only in Latvian-speaking schools, with resulting coverage of 51 percent of the desired population. Because coverage fell below the international 65 percent population-coverage criterion, Latvia is designated Latvia—LSS for Latvian-speaking schools.

Lithuania—Test administered only in Lithuanian-speaking schools, with resulting coverage of 84 percent of the desired population.

*Netherlands—Participation rate did not meet the international criteria of at least 50 percent participation by schools before replacement. The initial participation rate before replacement was 24 percent. The combined participation of schools and students was 60 percent.

*Romania—The pair of grades tested was one grade higher than the international target. Average student age in the upper grade was 14.6.

*Scotland—Participation rate did not meet the international criterion of 75 percent of schools and students combined. Participation rate was 73 percent after replacements for refusals were substituted.

*Slovenia—The pair of grades tested was one grade higher than the international target. Average student age was 14.8.

*South Africa—International guidelines requiring random selection of the classrooms to receive the assessment were not followed. Participation rate did not meet the international criterion of 75 percent of schools and students combined. Participation rate was 62 percent after replacements for refusals were substituted.

Switzerland—Test administered in 22 of 26 cantons, with resulting coverage of 86 percent of the desired population.

*Thailand—International guidelines requiring random selection of the classrooms to receive the assessment were not followed.

United States—Participation rate of 75 percent of schools and students combined was achieved only after replacements for refusals were substituted.

 

 

II. NAEP GRADE 8: STATES AND JURISDICTIONS WITH DEVIATIONS FROM PARTICIPATION RATE STANDARDS FOR 1996

In carrying out the 1996 state assessment program, the National Center for Education Statistics (NCES) established participation rate standards that jurisdictions were required to meet in order for their results to be reported. NCES also established additional standards that required the annotation of published results for jurisdictions whose sample participation rates were low enough to raise concerns about their representativeness.

Three states (Nevada, New Hampshire, and New Jersey) failed to meet the initial public school participation rate of 70 percent. For these states, results for grade 8 public school students are not reported in this or any report of NAEP 1996 findings. Several other jurisdictions whose results were published received a notation to indicate possible nonresponse bias.

NCES standards require weighted school participation rates before substitution of at least 85 percent to guard against potential bias due to school nonresponse. The NCES standards do not explicitly address the use of substitute schools to replace initially selected schools that declined to participate in the assessment. However, considerable technical consideration has been given to this issue. Even though the characteristics of the substitute schools were matched as closely as possible to the characteristics of the initially selected schools, substitution does not entirely eliminate the possibility of bias because of the nonparticipation of initially selected schools. Thus, for the weighted school participation rates that included substitute schools, the guideline was set at 90 percent. This is expressed in the following guideline:

A jurisdiction will receive a notation if its weighted participation rate for the initial sample of schools was below 85 percent and the weighted school participation rate after substitution was below 90 percent.

Seven jurisdictions did not meet this guideline for public schools at grade 8: Arkansas, Iowa, Michigan, Montana, New York, Vermont, and Wisconsin.

To help ensure adequate sample representation for each jurisdiction participating in the 1996 state assessment program, NAEP provided substitutes for nonparticipating schools. (When possible, a substitute school was provided for each initially selected school that declined participation.) For jurisdictions that used substitute schools, the assessment results were based on the student data from all schools participating from both the original sample and the list of substitutes (unless an initial school and its substitute eventually participated, in which case only the data from the initial school were used). For jurisdictions that did not use substitute schools, the participation rates were based on participating schools from the original sample.

The NCES standards specify that attention should be given to the representativeness of the sample coverage. Thus, inadequate representation of an important segment of a jurisdiction's population is of concern, regardless of the overall participation rate. A jurisdiction that is not already receiving a notation for problematic overall school or student participation rates will receive a notation if the sampled students within participating schools included a class of students with similar characteristics that had a weighted student response rate below 80 percent, and from which the nonresponding students together accounted for more than 5 percent of the jurisdiction's weighted assessable student sample.

At grade 8, Maryland and South Carolina (for public schools) failed to meet this NCES guideline.

In one state (Alaska), the public school student participation rate for grade 8 fell below the NCES-prescribed criterion of 85 percent.