International Requirements

In order to ensure comparability of the data across countries, the International Association for the Evaluation of Educational Achievement (IEA) established a set of detailed international requirements for the various aspects of data collection. The requirements regarding the target populations, sampling design, sample size, exclusions, and defining participation rates are described below.

International target populations
In order to identify comparable populations of students to be sampled, the IEA defined the international desired target population as follows:

• School population: all eligible schools¹ containing either one or more target grade students
• Eighth-grade student population: all students enrolled in the grade that represents the eighth year of schooling, counting from the first year of ISCED Level 1², provided that the mean age at the time of testing is at least 13.5 years. The target population of students in the United States corresponded to the eighth grade.
• Teacher population: all teachers teaching regular school subjects to students in the eighth grade (regardless of the subject or the number of hours taught) during the ICILS testing period and since the beginning of the school year.

Although participating education systems were expected to include all students in the International Target Population, sometimes it was not feasible to include all of these students because of geographic or linguistic constraints specific to the country or territory. Thus, each participating education system had its own “national” desired target population (also referred to as the National Target Population), which was the International Target Population reduced by the exclusions of those sections of the population that were not possible to assess. Working from the National Target Population, each participating education system had to operationalize the definition of its population for sampling purposes: i.e., define their “national” defined target population (referred to as the National Defined Population). While each education system’s National Defined Population ideally coincides with its National Target Population, in reality, there may be additional exclusions (e.g., of regions or school types) due to constraints of operationalizing the assessment (see section on Exclusions, below). In the United States there were no exclusions of this type, and therefore the National Defined Population and the National Target Population are the same.

Sampling design
It is not feasible to assess every eighth-grade student in each education system. Thus, a representative sample of eighth-grade students was selected. The sample design employed by the ICILS assessments are generally referred to as a two-stage stratified cluster sample. The sampling units at each stage were defined as follows.

• First-stage sampling units. In the first stage of sampling, sampling statisticians selected individual schools with a probability-proportionate-to-size (PPS) approach, which means that each school’s probability of selection is proportional to the estimated number of students enrolled in the target grade. Prior to sampling, sampling statisticians assigned schools in the sampling frame to a predetermined number of explicit or implicit strata. Then, sampling staff identified sample schools using a PPS systematic sampling method. Sampling statisticians also identified substitute schools (schools to replace original sampled school that refused to participate). As each school was selected in the sample, the two neighboring schools in the sampling frame, within the same stratum, were designated as substitute schools.
• Second-stage sampling units. In the second stage of sampling, The IEA Hamburg provided the Windows Within-School Sampling Software (WinW3S) for within-school sampling procedures. Schools provided a list of eighth-grade students and teachers of eighth-grade students. This involved sampling of students and teachers across target grade classes and target grade teachers in each sampled school. It should be noted that sampling with replacement of students within schools was not permitted.

Sample sizes
ICILS guidelines called for a minimum of 150 schools to be sampled, with 20 students and 15 teachers selected per school.

Exclusions
All schools and students excluded from the national defined target population are referred to as the excluded population. Exclusions could occur at the school level, with entire schools being excluded, or within schools, with specific students excluded. Some accommodations were made available for students with disabilities and for students who were unable to read or speak the language of the test. The IEA requirement with regard to exclusions is that they should not exceed more than 5 percent of the national desired target population.

School exclusions. Education systems could exclude schools that

• are geographically inaccessible;
• are of extremely small size;
• offer a curriculum, or school structure, radically different from the mainstream educational system; or
• provide instruction only to students in the excluded categories defined under within-school exclusions, such as schools for the blind.

Within-school exclusions. Education systems were instructed to adopt the following international within-school exclusion rules to define excluded students:

• Students with intellectual disabilities-These are students who are considered in the professional opinion of the school principal, or by other qualified staff members, to be intellectually disabled or who have been tested psychologically as such. This includes students who are emotionally or mentally unable to follow even the general instructions of the test. Students should not be excluded solely because of poor academic performance or normal disciplinary problems.
• Functionally disabled students-These are students who are permanently physically disabled in such a way that they cannot perform in the ICILS testing situation. Functionally disabled students who are able to respond should be included in the testing.
• Non-native language speakers-These are students who are unable to read or speak the language(s) of the test and would be unable to overcome the language barrier of the test. Typically, a student who has received less than 1 year of instruction in the language(s) of the test should be excluded, but this definition may need to be adapted in different countries.

Defined participation rates
In order to minimize the potential for response biases, the IEA developed participation or response rate standards that apply to all participating education systems and govern both whether or not a participating education system’s data are included in the ICILS international database as well as the way in which national statistics are presented in the international reports. These standards were set using composites of response rates at the school, student and teacher levels; moreover, response rates were calculated with and without the inclusion of substitute schools (selected to replace original sample schools refusing to participate). The decision about how to report data if the sampling procedures have not been followed will be made on a case-by-case basis.

The response rate standards take the following two forms, distinguished primarily by whether or not the school participation rate of 85 percent was met.

• Category 1: Met requirements. Participants who meet one of the following conditions are considered to have fulfilled the IEA requirements: an un-weighted school response rate without replacement of at least 85 percent (after rounding to the nearest whole percent) and an unweighted overall student/teacher response rate (after rounding) of at least 85 percent; a weighted school response rate without replacement of at least 85 percent (after rounding to the nearest whole percent) and a weighted overall student/teacher response rate (after rounding) of at least 85 percent; or the product of the (unrounded) weighted school response rate without replacement and the (unrounded) weighted overall student/teacher response rate of at least 75 percent (after rounding to the nearest whole percent).
• Category 2: Met requirements after substitutes. In the case of participants not meeting the category 1 requirements, a participating education system’s data are considered acceptable if the following requirements are met: either an unweighted or weighted school response rate without replacement of at least 50 percent (after rounding to the nearest percent) and has an unweighted school response rate with replacement of at least 85 percent (after rounding to the nearest whole percent) and an unweighted overall student/teacher response rate (after rounding) of at least 85 percent; a weighted school response rate with replacement of at least 85 percent (after rounding to nearest whole percent) and a weighted overall student/teacher response rate (after rounding) of at least 85 percent; or the product of the (unrounded) weighted school response rate with replacement and the (unrounded) weighted overall student/teacher response rate of at least 75 percent (after rounding to the nearest whole percent).

Schools with less than 50 percent student participation are considered nonrespondents.

Participants satisfying the category 1 standard are included in the international tabular presentations without annotation. Those able to satisfy only the category 2 standard are included as well but are annotated to indicate their response rate status. Participants that do not meet category 1 or 2, but that documentation that they followed sampling procedures appeared in a separate section of the report.

---

¹ Some sampled schools may be considered ineligible, for example if they are closed, no longer have students at the target grade, or do not contain any eligible students (all students would be excluded due to the reasons provided above).

² The ISCED was developed by the United Nations Educational, Scientific, and Cultural Organization (UNESCO) to facilitate the comparability of educational levels across countries. ISCED Level 1 begins with the first year of formal, academic learning (UNESCO 2011). In the United States ISCED Level 1 begins at grade 1.