The IEA provided detailed international requirements related to target populations, sampling design, sampling size, exclusions, assessment administration, and defining participation rates to ensure comparability of the data across participating education systems. Participating countries and education systems were obliged to follow these requirements. Details about each of these requirements are described below.
To identify comparable populations of students to be sampled, the IEA defined the target populations as follows:
School population. All eligible schools3 containing one or more 4th-grade classrooms.
Fourth-grade student population. The international desired target population is all students enrolled in the grade that represents the fourth year of formal schooling, counting from the first year of the International Standard Classification of Education (ISCED) Level 1,4 providing that the mean age at the time of testing is at least 9.5 years. For most countries, the target grade is grade 4, or its national equivalent. All students enrolled in the target grade, regardless of their age, belong to the international desired target population.
Teacher population. The target population is all language arts teachers linked to the selected students. Note that these teachers are not a representative sample of teachers within the education system. Rather, they are the teachers who teach a representative sample of students in grade 4 within the education system.
It was not logistically or fiscally feasible to assess every fourth-grade student in the United States. As is done in all participating countries, a representative sample of fourth-grade students was selected. The sample design employed by the PIRLS and ePIRLS 2016 assessments is generally referred to as a two-stage stratified cluster sample. The sampling units at each stage were defined as follows.
First-stage sampling units. In the first stage of sampling, statisticians selected individual schools with a probability proportionate to size (PPS) approach, which means that the probability is proportional to the estimated number of students enrolled in the target grade. Prior to sampling, statisticians assigned schools in the sampling frame to a predetermined number of explicit or implicit strata.5 Then, sampling staff sampled schools using a PPS systematic sampling method. Statisticians also selected substitution schools, which were selected to replace those that we originally sampled but refused to participate. The original and substitution schools were selected simultaneously.
Second-stage sampling units. In the second stage of sampling, classrooms were selected within sampled schools using a sampling software provided by the TIMSS and PIRLS International Study Center at Boston College. The software uses a sampling algorithm for selecting classes that standardized the class sampling across schools and assures that the class selection procedures are uniform across countries. Classrooms were selected from a list of eligible classrooms received by each school and data entered into the sampling software. The software was programmed to select one or two classrooms from each school. All students in sampled classrooms were selected for assessment. Each classroom selected for PIRLS was also selected for ePIRLS.
PIRLS guidelines call for a minimum of 150 schools to participate, with a minimum of 4,000 students assessed. The basic sample design of one to two classrooms per school was designed to yield a total sample of approximately 4,500 students per population. Classrooms with fewer than nine enrolled students were combined with other classrooms to create pseudo-classrooms prior to sampling. Thus, some schools may have up to three classrooms selected to participate in PIRLS.
The IEA defines exclusions at the school-level and at the student-level. PIRLS and ePIRLS 2016 did not provide accommodations for students with disabilities or students who were unable to read or speak the language of the test. The IEA requires that excluded students, including both the school-level and student-level, must not account for more than 5 percent of the national desired target population.
School exclusions. Countries could exclude schools that
Student exclusions. Countries were asked to adapt the following international within-school exclusion rules to define excluded students:
Students with functional disabilities—Students who are permanently physically disabled in such a way that they cannot perform in the PIRLS assessment situation. Students with functional disabilities who are able to respond were to be included in the testing.
Students with intellectual disabilities—Students who, in the professional opinion of the school principal or other qualified staff members, are considered to have intellectual disabilities or who have been tested psychologically as such. This includes students who are emotionally or mentally unable to follow even the general instructions of the test. Students were not to be excluded solely because of poor academic performance or normal disciplinary problems.
Non-native-language speakers—Students who are unable to read or speak the language(s) of the assessment and would be unable to overcome the language barrier of the assessment situation. Typically, a student who had received less than 1 year of instruction in the language(s) of the test was to be excluded.
The IEA requires that classrooms within sampled schools consisting entirely of students who belong to one of the exclusion categories be excluded prior to classroom sampling.
To minimize the potential for response biases, the IEA developed participation or response rate standards that apply to all participating education systems. These response rate standards govern whether an education system's data are included in the PIRLS 2016 international dataset and the way in which national statistics are presented in the international reports.
Response rates are calculated by taking into account school, classroom, and student participation, and are calculated with and without the use of replacement schools which were included as a substitute for schools refusing to participate. Response rates are defined as belonging in one of two categories.
Category 1: Met requirements. Education systems that meet all of the following conditions are considered to have fulfilled the IEA requirements: (a) a minimum school participation rate of 85 percent, based on original sampled schools only; and (b) a minimum classroom participation rate of 95 percent, from both original and substitute schools; and (c) a minimum student participation rate of 85 percent, from both original and substitute schools.
Category 2: Met requirements after substitutes. In the case of education systems not meeting the category 1 requirements, and as long as at least 50 percent of schools in the original sample participate, an education system's data are considered acceptable if the following requirements are met: a minimum combined school, classroom and student participation rate of 75 percent, based on the product of the participation rates described above. That is, the product of (a), (b), and (c), as defined in the category 1 standard, must be greater than or equal to 75 percent.
Education systems satisfying the category 1 standard are included in the international tabular presentations without annotation. Those able to satisfy only the category 2 standard are included as well but are annotated to indicate their response rate status. The data from education systems failing to meet either standard are presented separately in the international tabular presentations.
3 Some sampled schools may be considered ineligible for reasons noted in the section below titled "School exclusions."
4 The ISCED was developed by the United Nations Educational, Scientific, and Cultural Organization (UNESCO) to facilitate the comparability of educational levels across countries. ISCED Level 1 begins with the first year of formal, academic learning (UNESCO 2011). In the United States, ISCED Level 1 begins at grade 1.
5 The primary purpose of stratification is to improve the precision of the survey estimates. If explicit stratification of the population is used, the units of interest (schools, for example) are sorted into mutually exclusive subgroups–strata. Units in the same stratum are as homogeneous as possible, and units in different strata are as heterogeneous as possible, with respect to the characteristics of interest to the survey. Separate samples are then selected from each stratum. In the case of implicit stratification, the units of interest are simply sorted with respect to one or more variables known to have a high correlation with the variable of interest. In this way, implicit stratification guarantees that the sample of units selected will be spread across the categories of the stratification variables.