Sampling and Data Collection in the United States

Return to Methodology and Technical Notes

The PISA 2015 school sample was drawn for the United States by the PISA consortium. The U.S. PISA sample was stratified into 11 explicit groups based on region of the country (Northeast, Central, West, Southeast)³, control of school (public or private), and whether or not the school included the modal grade (grade 10). Within each stratum, the frame was sorted for sampling by five categorical stratification variables: grade range of the school (five categories); type of location relative to populous areas (city, suburb, town, rural);⁴ combined percentage of Black, Hispanic, Asian, Native Hawaiian/Pacific Islander, and American Indian/Alaska Native students (above or below 15 percent); gender (mostly female (percent female ≥ 95 percent), mostly male (percent female < 5 percent); and other); and state. The same frame and stratification variables were used for the Massachusetts and North Carolina samples. For the Puerto Rico sample, within each stratum, the frame was also sorted by district (Mayagüez, Bayamón, San Juan, Caguas, Humacao, Arecibo, Ponce).

For the U.S. national sample, the two U.S. states and Puerto Rico, within each school, 42 students aged 15 were randomly sampled. If fewer than 42 age-eligible students (in schools in either the national or subnational samples) were enrolled, all 15-year-old students in a school were selected. Thus, in each school, each age-eligible student had an equal probability of being selected. In order to be eligible for PISA, students had to be born between July 1, 1999, and June 30, 2000.

In the U.S. national, Massachusetts, and North Carolina samples, of the 42 students in each school who took the mathematics, science, and reading literacy assessments, 11 students were also subsampled to take the financial literacy assessment. The financial literacy assessment was held after the main PISA assessment, which required financial literacy students to return for a second session. The financial literacy assessment was administered in Massachusetts and North Carolina but not in Puerto Rico. The United States tested within the maximum testing period length of 42 days, from October to November 2015.

The U.S. PISA 2015 national school sample consisted of 240 schools.⁵ This number represents an increase from the international minimum requirement of 150 and was implemented to offset anticipated school nonresponse and reduce design effects. Schools were selected with probability proportionate to the school's estimated enrollment of 15-year-olds. The data for public schools were from the 2012–13 Common Core of Data (CCD) and the data for private schools were from the 2011–12 Private School Universe Survey (PSS). Any school containing at least one of grades 7 through 12was included in the school sampling frame. Participating schools provided a list of 15-year-old students (typically in August or September 2015) from which the sample was drawn using sampling software provided by the international contractor.

In addition to the international response rate standards described in the prior section, the U.S. sample had to meet the statistical standards of the National Center for Education Statistics (NCES) of the U.S. Department of Education. For an assessment like PISA, NCES requires that a nonresponse bias analysis be conducted when the response rate for schools falls below 85 percent or the response rate for students falls below 85 percent.

In order to keep PISA as inclusive as possible and to keep the exclusion rate down, the United States, Massachusetts, and North Carolina used the UH ('Une Heure') instrument designed for students with special education needs (Puerto Rico did not use the UH instrument). The UH instrument was available to special education needs students within mainstream schools and contained about half as many items as the regular test instrument. These testing items were deemed more suitable for students with special education needs. A UH student questionnaire was also administered, which only contained trend items from the regular student questionnaire. The timing structure of both the UH test instrument and UH student questionnaire allowed more time per question than the regular instruments and UH sessions were generally held in small groups.

---

³ The Northeast region consists of Connecticut, Delaware, the District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont. The Central region consists of Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, Wisconsin, and South Dakota. The West region consists of Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming. The Southeast region consists of Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia
⁴ These types are defined as follows: (1) "city" is a territory inside an urbanized area with a core population of 50,000 or more and inside a principal city; (2) "suburb" is a territory inside an urbanized area with a core population of 50,000 or more and outside a principal city; (3) "town" is a territory inside an urban cluster with a core population between 25,000 and 50,000; and (4) "rural" is territory not in an urbanized area or urban cluster.
⁵ The samples for Massachusetts and North Carolina consisted of 59 schools each; the sample for Puerto Rico consisted of 55 schools. As with the PISA national sample, these numbers were increased from the international minimum of 50 schools for subnational entities to offset anticipated school nonresponse and ineligibility.

Top