Methodology and Technical Notes - International Requirements

Return to Methodology and Technical Notes

OECD required all participating education systems to adhere to the PISA 2012 technical standards (OECD forthcoming), which provided detailed information about the target population, sampling, response rates, translation, assessment administration, and data submission. According to the standards, the international desired population in each education system consisted of 15-year-olds attending both publicly and privately controlled schools in grade 7 and higher. To provide valid estimates of student achievement and characteristics, the sample of PISA students had to be selected in a way that represented the full population of 15-year-old students in each education system. The sample design for PISA 2012 was a stratified systematic sample, with sampling probabilities proportional to the estimated number of 15-year-old students in the school based on grade enrollments. Samples were drawn using a two-stage sampling process. The first stage was a sample of schools, and the second stage was a sample of students within schools. The PISA international contractors responsible for the design and implementation of PISA internationally (hereafter referred to as the PISA consortium) drew the sample of schools for each education system.

A minimum of 4,500 students from a minimum of 150 schools was required in each country.¹ Following the PISA consortium guidelines, replacement schools were identified at the same time the PISA sample was selected by assigning the two schools neighboring the sampled school in the frame as replacements. The international guidelines specified that within schools, a sample of 35 students was to be selected in an equal probability sample unless fewer than 35 students age 15 were available (in which case all 15-year-old students were selected).

Each education system collected its own data, following international guidelines and specifications. The technical standards required that students in the sample be 15 years and 3 months to 16 years and 2 months at the beginning of the testing period. The maximum length of the testing period was 42 days. Most education systems conducted testing from March through August 2012.²

The school response-rate target was 85 percent for all education systems. This target applies in aggregate, not to each individual school. A minimum of 65 percent of schools from the original sample of schools was required to participate for an education system’s data to be included in the international database. Education systems were allowed to use replacement schools (selected during the sampling process) to increase the response rate once the 65 percent benchmark had been reached. Replacement students within a school were not allowed.

The technical standards also required a minimum participation rate of 80 percent of sampled students from schools (sampled and replacement) within each education system. Follow-up sessions were required in schools where too few students participated in the originally scheduled test sessions to ensure a high overall student response rate. A student was considered to be a participant if he or she participated in the first testing session or a follow-up or makeup testing session. Data from education systems not meeting this requirement could be excluded from international reports. See appendix B for final response rates by education system.

PISA 2012 is designed to be as inclusive as possible. The guidelines allowed schools to be excluded for approved reasons (for example, schools in remote regions, very small schools, or special education schools). Schools used the following international guidelines on student exclusions:

• Students with functional disabilities. These were students with a moderate to severe permanent physical disability such that they cannot perform in the PISA testing environment.
• Students with intellectual disabilities. These were students with a mental or emotional disability and who have been tested as cognitively delayed or who are considered in the professional opinion of qualified staff to be cognitively delayed such that they cannot perform in the PISA testing environment.
• Students with insufficient language experience. These were students who meet the three criteria of not being native speakers in the assessment language, having limited proficiency in the assessment language, and having less than 1 year of instruction in the assessment language.

Overall estimated exclusions (including both school and student exclusions) were to be under 5 percent of the PISA target population.