International Requirements for Sampling, Data Collection, and Response Rates

Return to Methodology and Technical Notes

OECD required all participating education systems(countries and subnational regions) to adhere to the PISA 2015 technical standards (OECD forthcoming), which provided detailed information about the target population, sampling, response rates, translation, assessment administration, and data submission. According to the standards, the international desired population in each education system consisted of 15-year-olds attending either publicly and privately controlled schools in grade 7 and higher. To provide valid estimates of student achievement and characteristics, the sample of PISA students had to be selected in a way that represented the full population of 15-year-old students in each education system. The sample design for PISA 2015 was a stratified systematic sample, with sampling probabilities proportional to the estimated number of 15-year-old students in the school based on grade enrollments. Samples were drawn using a two-stage sampling process. The first stage was a sample of schools, and the second stage was a sample of students within schools. The PISA international contractors responsible for the design and implementation of PISA internationally (hereafter referred to as the PISA consortium) drew the sample of schools for each education system.

A minimum of 5,400 students from a minimum of 150 schools was required in each country that planned to administer computer-based assessments.¹ Following the PISA consortium guidelines, replacement schools were identified at the same time the PISA sample was selected by assigning the two schools neighboring the sampled school in the frame as replacements. The international guidelines specified that within schools, a sample of 42 students was to be selected in an equal probability sample unless fewer than 42 students age 15 were available (in which case all 15-year-old students were selected).

Each education system collected its own data, following international guidelines and specifications. The technical standards required that students in the sample be 15 years and 3 months to 16 years and 2 months at the beginning of the testing period (hereafter referred to as "15-year-olds" or "15-year-old students"). The maximum length of the testing period was consecutive 42 days. Most education systems conducted testing from March through August 2015.²

The school response-rate target was 85 percent for all education systems. A minimum of 65 percent of schools from the original sample of schools was required to participate for an education system's data to be included in the international database. Education systems were allowed to use replacement schools (selected during the sampling process) to increase the response rate once the 65 percent benchmark had been reached.

The technical standards also required a minimum participation rate of 80 percent of sampled students from schools (sampled and replacement) within each education system. This target applied in aggregate, not to each individual school. Follow-up sessions were required in schools where too few students participated in the originally scheduled test sessions to ensure a high overall student response rate. Replacement students within a school were not allowed. A student was considered to be a participant if he or she participated in the first testing session or a follow-up or makeup testing session. Data from education systems not meeting this requirement are excluded from international reports.

PISA 2015 is designed to be as inclusive as possible. The guidelines allowed schools to be excluded for approved reasons (for example, schools in remote regions, very small schools, or special education schools). Schools used the following international guidelines on student exclusions:

Students with functional disabilities. These are students with a moderate to severe permanent physical disability such that they cannot perform in the PISA testing environment.

Students with intellectual disabilities. These are students with a mental or emotional disability and who have been tested as cognitively delayed or who are considered in the professional opinion of qualified staff to be cognitively delayed such that they cannot perform in the PISA testing environment.

Students with insufficient language experience. These are students who meet the three criteria of not being native speakers in the assessment language, having limited proficiency in the assessment language, and having less than 1 year of instruction in the assessment language.

Overall estimated exclusions (including both school and student exclusions) were to be under 5 percent of the PISA target population. To keep PISA as inclusive as possible and to keep the exclusion rate down, the United States, Massachusetts, and North Carolina used the UH ('Une Heure') instrument designed for students with special education needs (Puerto Rico did not use the UH instrument). See the description of the UH instrument in the next section.

---

¹ Education systems that opted to conduct paper-based assessments were required to assess a minimum of 4,500 students from a minimum of 150 schools. PISA also includes education systems that are not countries, such as Hong Kong and Macao in China. Subnational entities such as these were required to sample a minimum of 1,500 students from at least 50 schools. In the United States, two states (Massachusetts, North Carolina) and Puerto Rico provided samples in addition to the schools for the national sample in order to obtain PISA estimates for their jurisdictions.
² The United States and the United Kingdom were given permission to move the testing dates to September through November in an effort to improve response rates. The range of eligible birth dates was adjusted so that the mean age remained the same (i.e., 15 years and 3 months to 16 years and 2 months at the beginning of the testing period). In 2003, the United States conducted PISA in the spring and fall and found no significant difference in student performance between the two time points.