Program for the International Assessment of Adult Competencies (PIAAC)

1. OVERVIEW

The Program for the International Assessment of Adult Competencies (PIAAC) is a cyclical household study that has been developed under the auspices of the Organization for Economic Cooperation and Development (OECD). The first cycle of PIAAC consisted of three rounds of data collection between 2011 and 2017. PIAAC's second cycle is planned to begin in 2021. For cycle one's initial study in 2011-12 ("round 1"), adults were surveyed in 24 participating countries. Nine additional countries participated in PIAAC in round 2 in 2014, and five more countries participated in PIAAC in round 3 in 2017.

The United States conducted three rounds of data collection during PIAAC first cycle. The initial U.S. study for round 1 was conducted from August 2011 through April 2012 with a nationally representative sample of 5,000 adults between the ages of 16 and 65 (officially known as the U.S. PIAAC Main Study). In 2013–14, the United States conducted a second round of data collection in order to (1) supplement the 2012 U.S. household data collection (officially known as the "National Supplement") and (2) include individuals who were detained in state, federal, or private prisons housing state or federal inmates in the United States (officially known as part of the National Supplement, but having its own name: the "U.S. PIAAC Prison Study"). The household supplement was conducted from August 2013 through April 2014 with a sample of 3,660 adults in households in order to enhance the U.S. PIAAC 2011-12 dataset by (a) oversampling young adults (age 16-34), (b) oversampling unemployed adults (age 16-65), and (c) expanding the sample to include older adults (age 66-74). The U.S. PIAAC Prison Study was conducted from February through June 2014 with a nationally representative sample of 1,270 adult inmates (age 18-74) who were detained in 98 states and federal prisons in the United States. The third round of U.S. data collection was administered from March 2017 to November 2017 to a nationally representative household sample of about 3,660 adults between the ages of 16-74. These data, released in the fall of 2019, provide a second point in time to allow comparisons with the 2012/14 results. In addition, the combined 2017 and 2012/2014 data support the calculation of small area estimates at the county and state level, which were released in April, 2020.

The goal of PIAAC is to assess and compare the basic skills and the broad range of competencies of adults around the world. The assessment focuses on cognitive and workplace skills needed for successful participation in 21st-century society and the global economy. Specifically, PIAAC measures relationships between individuals' educational background, workplace experiences and training, skill-used at work and home, occupational attainment, income, health, use of information and communications technology, and cognitive skills in the areas of literacy, numeracy, and digital problem solving.

PIAAC is a complex assessment: the data collection was conducted in multiple languages and in numerous countries with diverse populations, cultures, education and life experiences. However, in order to make results comparable all participating countries (a) follow the quality assurance guidelines set by the OECD Consortium, (b) closely follow all the agreed-upon standards set for survey design, implementation of the assessment and the reporting of results, and (c) administer equivalent versions of all cognitive and non-cognitive instruments.

PIAAC builds on knowledge and experiences gained from previous international adult assessments including the International Adult Literacy Survey (IALS) and the Adult Literacy and Lifeskills Survey (ALL). PIAAC enhances and expands on these previous assessments’ frameworks and, at the same time, improves upon their design and methodologies.

In the United States, the PIAAC assessment was conducted in English only; however, the PIAAC survey background questions were in both English and Spanish.

Purpose

The primary objectives of PIAAC are to (1) identify and measure cognitive competencies believed to underlie both personal and societal success, (2) assess the impact of these competencies on social and economic outcomes at individual and aggregate levels, (3) gauge the performance of education and training systems in generating required competencies, and (4) help to clarify the policy levers that could contribute to enhancing competencies.

One of PIAAC’s core objectives is to assess how well participants use information and communications technology to access, manage, integrate and evaluate information; construct new knowledge; and communicate with other people. In addition, PIAAC collected information on participants’ use of key work skills in their jobs, a first time for an international study. In this way, PIAAC offers a more complete and nuanced picture of human capital than earlier studies.

It is important that the participating countries share a set of survey objectives, to facilitate comparisons of survey results between countries. PIAAC assessment items and background questionnaires were designed to ensure cross–cultural, cross–national and cross–language validity.

Top

Components

In cycle one, PIAAC collected background information on adults before administering direct assessments of literacy, numeracy, reading components, and problem solving in technology-rich environments. After the interviewer-directed background questionnaire was administered, the PIAAC assessment design was to route participants to the appropriate delivery mode to assure the most reliable, valid, and comparable assessment of skills. Two delivery modes for the assessments were available: paper-and-pencil and computer-based. The computer-based mode was administered to participants who met all of the following criteria, established for determining basic computer competence: (1) prior computer use, (2) willingness to take the assessment on the computer, and (3) passing a basic computer test (by successfully completing four of six simple tasks, such as using a mouse and highlighting text on the screen). Participants who did not meet any one of these requirements - who reported no computer use, who were unwilling to take the assessment on the computer, or who failed the basic computer test - were routed to the paper-based assessment.

The percentage of U.S. adults routed to the paper-based assessment varied in each of the three rounds of U.S. data collection: in round 1 (2011-12), approximately 16 percent of U.S. adults in the household study received the paper-based assessment in round 2 (2014), approximately 22 percent of adults in the household study and 36 percent of adults in the prison study; and in round 3 (2017), about 16 percent of adults in the household study The same survey procedures, processes, and assessment instruments were used in all three rounds.

Background Questionnaire for the Household Studies. The background questionnaire for cycle one was developed to identify (a) what skills participants use in their job and in their home life, (b) how participants acquire those skills, and (c) how those skills are distributed throughout the population. Interviewers administered the background questionnaire, asking participants about their education and training; present and past work experience; the skills they use at work; their use of specific literacy, numeracy, and information and technology (ICT) skills at work and at home; personal traits; and demographic information.

In order to obtain background information from a wide range of respondents in the United States, the background questionnaire was administered in either English or Spanish (although the direct assessment of skills was only administered in English). For linking purposes, several items from IALS and ALL were included in the PIAAC background questionnaire.

Participating countries were allowed to add up to 5 minutes of country-specific items. The United States added questions focused on country of origin, language, race and ethnicity, training courses, and health related information. A majority of the U.S. country-specific questions were adopted from the 2003 National Assessment of Adult Literacy (NAAL) background questionnaire.

The household background questionnaire was identical for all three rounds of U.S. data collection in cycle one, except that in rounds 2 and 3 a few additional U.S. country-specific items were administered that were not in round 1.

Background Questionnaire for the Prison Study. While maintaining comparability between household and prison instruments as closely as possible, adaptations to the household questionnaire were implemented in order to adequately capture information from the prison population. These included deleting questions from the household questionnaire that would be irrelevant to respondents in prison, such as questions on earnings from their current job, as well as the addition of questions that addressed respondents’ specific activities in prison, such as participation in academic programs and English as a Second Language (ESL) classes offered in prison; experiences with prison work assignments; involvement in non-academic programs, such as life skills and employment readiness classes; and educational attainment and employment prior to incarceration.

PIAAC Assessment Tasks. PIAAC both paper- and computer-based assessment tasks were drawn from real-life situations that are expected to be of importance or relevance in different contexts. Tasks' contents and questions were intended to reflect the purposes of adults' daily lives across cultures, even if they were not necessarily familiar to all adults in all countries. While the background questionnaire was administered in English or Spanish, both paper- and computer-based assessments were administered in English only. Some assessment items from IALS and ALL were included in the PIAAC assessment to facilitate the linking between the three international assessments.

Paper–based Assessment. The paper-based assessment (PBA) began with a 10-minute core of literacy/ numeracy items in paper- and- pencil format. Participants who performed at or above a minimum standard on this core section were randomly assigned to either a 30-minute cluster of literacy items or a 30-minute cluster of numeracy items.  After they completed those items, they received a 20-minute assessment of reading components. Participants who performed poorly on the paper literacy/numeracy core proceeded directly to the reading components booklet.

Top

Computer–based Assessment. The PIAAC computer-based assessment (CBA) was only administered to participants who indicated having previous experience with computers in the background questionnaire interview and were willing to take the assessment on the computer. Participants were directed to a core section that was composed of two parts: a basic computer test (ICT core ), which measures skills such as highlighting, and a literacy/numeracy core which measures basic skills within these domains. Participants who performed poorly in either of these core tests were switched over to the appropriate sections of the paper-and-pencil instruments. Participants who failed the ICT core proceeded to the paper-based assessment and took the paper-based literacy/numeracy core items.  Participants passing the ICT core proceeded to the computer-based literacy/numeracy core. If they did not pass the computer-based literacy/numeracy core, participants were routed directly to the reading components section of the PBA.

Participants who performed well on both parts of the computer-based core section were randomly routed to the computer-based literacy, computer-based numeracy, or digital problem-solving domains.

Adaptive Design

One of the unique aspects of PIAAC was the adaptive design of the computer-based assessment (CBA) within the domains of literacy and numeracy. The adaptive testing process meant that respondents were directed to a set of easier or more difficult items. The choice of assessment items for each participant mainly depended on an algorithm using a set of variables that included (1) the participant’s level of education; (2) the participant’s status as a native or non-native English-language speaker; and (3) the participant’s performance in the CBA core (as well as their performance in the CBA module as they advance through the assessment). For the digital problem-solving domain, there was no adaptive process.

The key advantage of an adaptive design is to provide a more accurate assessment of participants’ abilities, while using a smaller number of assessment items than a traditional test design in which respondents must answer all questions included in the test, from easiest to most difficult.

Periodicity

Internationally, PIAAC has been envisaged as a decennial survey. The PIAAC cycle 1 assessment in the United States was conducted between 2011 and 2017 with three rounds of data collection. Internationally, three rounds of data collection involved an expansion of the original 24 countries to an additional 14 countries in the subsequent rounds. PIAAC cycle 2 will be administered in 2021-22, with currently 33 countries, including the United States, taking part in the development.

Data Availability

Information on the availability of data and schedule of releases for PIAAC can be found at https://nces.ed.gov/surveys/piaac/schedule_detailed.asp

Top