Program for International Student Assessment (PISA): 2012 Results -Methodology and Technical Notes - Sampling and Data Collection in the United States

Back to PISA 2012 Results

Methodology and Technical Notes - Sampling and Data Collection in the United States

Return to Methodology and Technical Notes

The PISA 2012 school sample was drawn for the United States by the PISA consortium. The U.S. PISA sample was stratified into eight explicit groups based on control of school (public or private) and region of the country (Northeast, Central, West, Southeast).¹ Within each stratum, the frame was sorted for sampling by five categorical stratification variables: grade range of the school (five categories); type of location relative to populous areas (city, suburb, town, rural);² combined percentage of Black, Hispanic, Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native students (above or below 15 percent); gender (mostly female [percent female >= 95 percent], mostly male [percent female < 5 percent]; and other); and state. The same frame and characteristics were used for the state samples.

For the U.S. national sample, within each school, 50 students aged 15 were randomly sampled. The United States increased its national sample from the international standard of 35 to 50 in order to reach the required number of students and in order to administer the optional financial literacy assessment.Connecticut, Florida, and Massachusetts participated in PISA 2012 with separate state samples drawn by the PISA consortium. The state samples are not part of the main sample. In each of the three state samples, 42 students aged 15 were randomly sampled within each school. If fewer than 50 age-eligible students (in schools in the national sample) or fewer than 42 age-eligible students (in schools in the state samples) were enrolled, all 15-year-old students in a school were selected. Thus, in each school, each age-eligible student had an equal probability of being selected. Sampled students were born between July 1, 1996, and June 30, 1997 (hereafter the sampled students are referred to as “15-year-olds” or “15-year-old students”). Sampled students mainly came from three grades—grades 9, 10, and 11. Table AA1 provides the grade distribution of U.S. 15-year-old students.

Table AA1. Percentage distribution of U.S. 15-year-old students, by grade level: 2012
Grade level	Percent	*s.e.*
Grade 7	#	†
Grade 8	‡	†
Grade 9	11.7	1.06
Grade 10	71.2	1.10
Grade 11	16.6	0.83
Grade 12	‡	†
Total	100.0
# Rounds to zero. † Not applicable. ‡ Reporting standards not met. NOTE: Standard error is noted by s.e. SOURCE: Organization for Economic Cooperation and Development (OECD), Program for International Student Assessment (PISA), 2012.

In the national-sample schools, of the 50 students, 42 took the paper-based mathematics, science, and reading literacy assessments, 20 of which were subsampled to also take the computer-based assessment, and 8 took the financial literacy assessment. In the state sample schools, all sampled students took only the paper-based mathematics, science, and reading assessments. The technical standard for the maximum length of the testing period was 42 days, but the United States requested and was granted permission to expand the testing window to 60 days (from October 2, 2012, to November 30, 2012) to accommodate school requests.

The U.S. PISA 2012 national school sample consisted of 240 schools.³ This number was increased from the international minimum requirement of 150 to offset school nonresponse and reduce design effects. Schools were selected with probability proportionate to the school’s estimated enrollment of 15-year-olds. The data for public schools were from the 2008–09 Common Core of Data and the data for private schools were from the 2009–10 Private School Universe Survey. Any school containing at least one 7th- through 12th-grade class was included in the school sampling frame. Participating schools provided a list of 15-year-old students (typically in August or September 2012) from which the sample was drawn using sampling software provided by the international contractor.

In addition to the international response rate standards described in the prior section, the U.S. sample had to meet the statistical standards of the National Center for Education Statistics (NCES) of the U.S. Department of Education. For assessments, NCES requires that the response rate should be at least 80 percent for schools and at least 85 percent for students.

¹ The Northeast region consists of Connecticut, Delaware, the District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont. The Central region consists of Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, Wisconsin, and South Dakota. The West region consists of Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming. The Southeast region consists of Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia.

² These types are defined as follows: (1) “city” is a territory inside an urbanized area with a core population of 50,000 or more and inside a principal city; (2) “suburb” is a territory inside an urbanized area with a core population of 50,000 or more and outside a principal city; (3) “town” is a territory inside an urban cluster with a core population between 25,000 and 50,000; and (4) “rural” is a territory not in an urbanized area or urban cluster.

³ The state samples consisted of 54, 55, and 54 schools for Connecticut, Florida, and Massachusetts, respectively. As with the PISA national sample, these numbers were increased from the international minimum of 50 schools for subnational entities to offset school nonresponse and ineligibility.