Skip Navigation

International Requirements for Sampling, Data Collection, and Response Rates

Back

In order to ensure comparability of the data across countries, the IEA established a set of detailed international requirements for the various aspects of data collection. The requirements regarding the target populations, sampling design, sample size, exclusions, and defining participation rates are described below.

International Target Populations
In order to identify comparable populations of students to be sampled, the IEA defined the international desired target population as follows:

  • School population: all eligible schools1 containing either one or more 4th-grade classrooms, one or more 8th-grade classrooms or one or more 12th-grade students.
  • Teacher population: all mathematics and science teachers who teach the selected students in grades 4 and 8. Note that these teachers are not a representative sample of teachers within the country. Rather, they are the mathematics and science teachers who teach a representative sample of students in two grades within the country (grades 4 and 8 in the United States). Similarly, the target population for TIMSS Advanced is all advanced mathematics and physics teachers who teach the selected advanced students.
  • Fourth-grade student population: all students enrolled in the grade that represents the 4th year of schooling, counting from the first year of the International Standard Classification of Education (ISCED) Level 12 provided that the mean age at the time of testing is at least 9.5 years.
  • Eighth-grade student population: all students enrolled in the grade that represents the 8th year of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing is at least 13.5 years.
  • TIMSS Advanced student population for advanced mathematics: all students in the final year of secondary schooling who have taken or are taking courses in advanced mathematics. The courses that define the target populations have to cover most, if not all, of the advanced mathematics topics that were outlined in the TIMSS Advanced 2015 Assessment Frameworks.
  • TIMSS Advanced student population for physics: all students in the final year of secondary schooling who have taken or are taking courses in physics. As with advanced mathematics, the courses that define the target populations have to cover most, if not all, of the physics topics that were outlined in the TIMSS Advanced 2015 Assessment Frameworks.

In some countries the TIMSS Advanced student populations are 100% overlapping, and in others, like the United States, they are overlapping to a variable degree.

The IEA TIMSS Advanced 2015 Assessment Framework (available at http://timss.bc.edu/timss2015-advanced/frameworks.html) specifies, at the highest level, that the target population includes students who have taken courses that contain the following content, respectively:

  • Advanced mathematics content
    • Algebra
    • Geometry
    • Calculus
  • Physics content
    • Mechanics and thermodynamics
    • Electricity and magnetism
    • Wave phenomena and atomic/nuclear physics

Although participating education systems were expected to include all students in the International Target Population, sometimes it was not feasible to include all of these students because of geographic or linguistic constraints specific to the country or territory. Thus, each participating education system had its own "national" desired target population (also referred to as the National Target Population), which was the International Target Population reduced by the exclusions of those sections of the population that were not possible to assess. Working from the National Target Population, each participating education system had to operationalize the definition of its population for sampling purposes: i.e., define their "national" defined target population (referred to as the National Defined Population). While each education system's National Defined Population ideally coincides with its National Target Population, in reality, there may be additional exclusions (e.g., of regions or school types) due to constraints of operationalizing the assessment (see section on Exclusions, below). In the United States there were no exclusions of this type.

Sampling design
It is not feasible to assess every 4th-, 8th and, advanced 12th-grade student in each education system. Thus, a representative sample of 4th-, 8th and advanced 12th-grade students was selected. The sample design employed by the TIMSS and TIMSS Advanced assessments are generally referred to as a two-stage stratified cluster sample. The sampling units at each stage were defined as follows.

  • First-stage sampling units. In the first stage of sampling, sampling statisticians selected individual schools with a probability-proportionate-to-size (PPS) approach, which means that each school's probability of selection is proportional to the estimated number of students enrolled in the target grade. Prior to sampling, sampling statisticians assigned schools in the sampling frame to a predetermined number of explicit or implicit strata. Then, sampling staff identified sample schools using a PPS systematic sampling method. Sampling statisticians also identified substitute schools (schools to replace original sampled school that refused to participate). As each school was selected in the sample, the two neighboring schools in the sampling frame, within the same stratum, were designated as substitute schools.
  • Second-stage sampling units. In the second stage of sampling at grades 4 and 8, sampling statisticians selected classrooms within sampled schools using sampling software provided by the TIMSS & PIRLS International Study Center at Boston College. The software uses a sampling algorithm for selecting classes that standardized the class sampling across schools and assures uniformity in the class selection procedures across participants. The software identified at least one classroom from a list of eligible classrooms that sampling staff prepared for each target grade. In various countries and education systems, including the United States, more than one eligible classroom per target grade per school was selected when possible. All students in sampled classrooms were selected for assessment.

In the second stage of sampling of the TIMSS Advanced sample, the sampling statisticians selected students rather than classrooms within sampled schools only for the United States. The methodology, described in more detail below, was designed specifically for the United States target population and to meet international guidelines described next. There are two reasons students were sampled instead of classrooms: (1) students did not have to be enrolled in an eligible class at the time of sampling (they could have taken an eligible class in a previous grade), so sampling classrooms would miss eligible students; and (2) advanced mathematics and physics student populations are overlapping to a variable degree, which means sampling classrooms would have been inefficient and the required number of assessed students in each subject would have been more difficult to achieve.

Sample sizes

Fourth- and eighth-grades. TIMSS guidelines called for a minimum of 150 schools to be sampled per grade, with a minimum of 4,000 students assessed per grade. The basic sample design of one classroom per target grade per school was designed to yield a total sample of approximately 4,500 students per population. Countries with small class sizes or fewer than 30 students per school were directed to consider sampling more schools, more classrooms per school, or both, to meet the minimum target of 4,000 tested students.

TIMSS Advanced. TIMSS guidelines called for a minimum of 120 schools to be sampled, with a minimum of 3,600 students assessed per subject. If classes were selected at the second stage of sampling, at least one classroom/group from the target population was selected in each sampled school. If students were directly selected at the second stage of sampling, 30 advanced mathematics students and 30 physics students were selected within each school, provided the sample size requirements were met; otherwise, the number of selected students may be larger.

Exclusions

The following discussion draws on the TIMSS 2015 Survey Operations Procedures Unit 1: School Sampling and TIMSS Advanced 2015 Survey Operations Procedures Unit 1: School Sampling. All schools and students excluded from the national defined target population are referred to as the excluded population. Exclusions could occur at the school level, with entire schools being excluded, or within schools, with specific students or entire classrooms excluded. In 2015 for TIMSS and TIMSS Advanced, some accommodations were made available for students with disabilities and for students who were unable to read or speak the language of the test. The IEA requirement with regard to exclusions is that they should not exceed more than 5 percent of the national desired target population.

School exclusions. Education systems could exclude schools that

  • are geographically inaccessible;
  • are of extremely small size;
  • offer a curriculum, or school structure, radically different from the mainstream educational system; or
  • provide instruction only to students in the excluded categories defined under “within-school exclusions,” such as schools for the blind.

Within-school exclusions. Education systems were instructed to adopt the following international within-school exclusion rules to define excluded students:

  • Students with intellectual disabilities—Students who, in the professional opinion of the school principal or other qualified staff members, are considered to have intellectual disabilities or who have been tested psychologically as such. This includes students who are emotionally or mentally unable to follow even the general instructions of the test. Students were not to be excluded solely because of poor academic performance or normal disciplinary problems.
  • Students with functional disabilities—Students who are permanently physically disabled in such a way that they cannot perform in the TIMSS testing situation. Students with functional disabilities who are able to respond were to be included in the testing.
  • Non-native-language speakers—Students who are unable to read or speak the language(s) of the test and would be unable to overcome the language barrier of the test. Typically, a student who had received less than one year of instruction in the language(s) of the test was to be excluded.

Defined participation rates

In order to minimize the potential for response biases, the IEA developed participation or response rate standards that apply to all participating education systems and govern both whether or not a participating education system's data are included in the TIMSS and TIMSS Advanced international database as well as the way in which national statistics are presented in the international reports. These standards were set using composites of response rates at the school, classroom3 , and student and teacher levels; moreover, response rates were calculated with and without the inclusion of substitute schools (selected to replace original sample schools refusing to participate).

The response rate standards take the following two forms, distinguished primarily by whether or not the school participation rate of 85 percent was met.

  • Category 1: Met requirements. Participants that meet all of the following conditions are considered to have fulfilled the IEA requirements: (a) a minimum school participation rate of 85 percent, based on original (sampled) schools only; (b) a minimum classroom participation rate of 95 percent, from both original and substitute schools; and (c) a minimum student participation rate of 85 percent, from both original and substitute schools.
  • Category 2: Met requirements after substitutes. In the case of participants not meeting the category 1 requirements, a participating education system's data are considered acceptable if the following requirements are met: a minimum combined school, classroom, and student participation rate of 75 percent, based on the product of the participation rates described above. That is, the product of (a), (b), and (c), as defined in the category 1 standard, must be greater than or equal to 75 percent.

Classrooms with less than 50 percent student participation are considered non-respondents and are dropped from reporting by the IEA's Data Processing Center (DPC).

Participants satisfying the category 1 standard are included in the international tabular presentations without annotation. Those able to satisfy only the category 2 standard are included as well but are annotated to indicate their response rate status.


1 Some sampled schools may be considered ineligible for reasons noted in the exclusions” section below. For TIMSS Advanced, eligibility was also determined by having 12th-grade students who had taken or were taking one or more of the eligible advanced mathematics or physics courses.

2 The ISCED was developed by the United Nations Educational, Scientific, and Cultural Organization (UNESCO) to facilitate the comparability of educational levels across countries. ISCED Level 1 begins with the first year of formal, academic learning (UNESCO 2011). In the United States, ISCED Level 1 begins at grade 1.

3 Because there was no classroom sampling for TIMSS Advanced in the United States, the classroom rate is 100 percent.
Back