In addition to the following questions about PIAAC, more FAQs about international assessments are available at: http://nces.ed.gov/surveys/international/faqs.asp.
PIAAC is designed to assess adults over a broad range of abilities—from simple reading to complex computer-based problem solving skills. All countries that participated in PIAAC in 2012 assessed the domains of literacy and numeracy in both a paper-and-pencil mode and a computer-administered mode. In addition, some countries assessed problem solving administered on a computer as well as components of reading (administered only in paper-and-pencil format). The U.S. assessed all four domains.
The assessment was designed to be valid cross-culturally and cross-nationally.
PIAAC assessment questions are developed in a collaborative, international process. PIAAC assessment questions were based on frameworks developed by internationally-known experts in each subject or domain. Assessment experts and developers from Ministries/Departments of Education and Labor and OECD staff participated in the conceptualization, creation, and extensive yearlong reviews of assessment questions. In addition, the PIAAC Consortium's support staff, assisted by expert panels, researchers, and working groups, developed the assessment Background Questionnaire (BQ). The PIAAC Consortium also guided the development of common standards and procedures for collecting and reporting data, as well as an international "virtual machine" software that administers the assessment uniformly across countries. All PIAAC countries follow the common standards and procedures and use the virtual machine software when conducting the survey and assessment. As a result, PIAAC can provide a reliable and comparable measure of literacy skills in the adult population of participating countries.
Test items, instruments, and procedures are field-tested prior to administration. Before the administration of the assessment, a field test was conducted in the participating countries. The PIAAC Consortium analyzed the field test data and implemented changes to eliminate problematic items or revise procedures prior to the administration of the assessment.
The design and implementation of PIAAC was guided by technical standards and guidelines developed by literacy experts to ensure that the survey yielded high-quality and internationally comparable data. For example, for their survey operations, participating countries were required to develop a quality assurance and quality control program that included information about the design and implementation of PIAAC data collection. In addition, all countries were required to adhere to recognized standards of ethical research practices in regard to respect for respondent privacy and confidentiality, the importance of ethics and scientific rigor in research involving human subjects, and avoiding practices or methods that may harm or seriously mislead survey participants. Compliance with the technical standards was mandatory and monitored throughout the development and implementation phases of the data collection through direct contact, submission of evidence that required activities were completed, and on-going collection of data from countries concerning key aspects of implementation.
In addition, participating countries provided standardized training to the interviewers who administered the assessment in order to familiarize them with the survey procedures that would allow them to administer the assessment consistently across respondents and reduce the potential for erroneous data. After the data collection process, the quality of each participating country's data was reviewed prior to publication. The review was based on the analysis of the psychometric characteristics of the data and evidence of compliance with the technical standards.
The problem solving in technology-rich environments domain assesses the cognitive processes of problem solving: goal setting, planning, selecting, evaluating, organizing, and communicating results. In a digital environment, these skills involve understanding electronic texts, images, graphics and numerical data, as well as locating, evaluating, and critically judging the validity, accuracy, and appropriateness of the accessed information.
The environment in which PIAAC problem solving is assessed is meant to reflect the fact that digital technology has changed the ways in which individuals live their day-to-day lives, communicate with others, work, conduct their affairs, and access information. The information and communication technology tools such as computer applications, the Internet, and mobile technologies are all part of the environments in which individuals operate. In PIAAC, items for problem solving in technology-rich environments are presented on laptop computers in simulated software applications using commands and functions commonly found in email, web browsers, and spreadsheets.
Countries that participate in PIAAC must draw a sample of individuals ages 16 to 65 that represent the entire population of adults living in households in the country. Some countries draw their samples from their national registries of all persons in the country; others draw their samples from census data. In the United States, a nationally representative household sample was drawn from the most current Census Bureau population estimates.
The U.S. sample design employed by the PIAAC is generally referred to as a four-stage stratified area probability sample. This method involves the selection of (1) primary sampling units (PSUs) consisting of counties or groups of contiguous counties, (2) secondary sampling units (referred to as segments) consisting of area blocks, (3) dwelling units (DUs), and (4) eligible persons (ultimate sampling unit) within DUs. Random selection methods are used, with calculable probabilities of selection at each stage of sampling. This sample design ensures the production of reliable statistics for a minimum of 5,000 completed cases.
The PIAAC main study's target population consisted of non-institutionalized adults age 16 to 65 who resided in the United States at the time of survey, where age was determined as part of an initial screener questionnaire. Adults were included regardless of citizenship, nationality, or language. The main study's target population included only persons living in households or group quarters; it excluded all other persons (such as persons living in shelters, the incarcerated, military personnel who live in barracks or bases, or persons who live in institutionalized group quarters, such as hospitals or nursing homes). The target population included full-time and part-time members of the military who did not reside in military barracks or military bases, adults in other non-institutional collective dwelling units, such as workers' quarters or halfway homes, and adults who lived at school in student group quarters, such as a dormitory, fraternity or sorority.
In 2013-14, the U.S. PIAAC National Supplement repeated the administration of PIAAC with a target population of 3,600 adults. It used the same procedures, instruments, and assessments that were used for the PIAAC Main Study (2011-12). The National Supplement increased the sample size of key U.S. subgroups of interest, including unemployed adults (ages 16–65), young adults (ages 16–34), and older adults (ages 66–74). A separate Prison Study extended the PIAAC assessment to incarcerated adults in the United States. The Prison Study sample included 1,200 inmates (ages 16–74) in state, federal, or private prisons in the United States.
All adults, regardless of immigration status, were part of the main study's target population for the assessment. In order to get a representative sample of the adult population currently residing in the United States, respondents were not asked about citizenship status before taking the assessment and were guaranteed anonymity for all their answers to the background questionnaire. Although the assessment was only administered in English, the background questionnaire was offered in both Spanish and English. These procedures allowed the estimates to be applicable to all adults in the United States, regardless of citizenship or legal status, and they mitigated the effects of low-English language proficiency. Non-native born adults had, on average, lower scores than native-born adults. This was true in most participating countries. The percentage of non-native-born adults in the U.S. was 15 percent. The average percentage of non-native-born adults across all participating countries was 12 percent, and ranged from 28 percent in Australia to less than 1 percent in Japan.
Sampling is carefully planned and monitored. The rules of participation require that countries design a sampling plan according to the standards provided in the PIAAC Technical Standards and Guidelines and submit their plans to the PIAAC Consortium for approval. In addition to a sampling plan, countries were required to complete quality control forms to verify that their sample was selected in an unbiased and randomized way. Quality checks were performed by the PIAAC Consortium to ensure that submitted sampling plans were accurately followed.
No, PIAAC is a voluntary assessment.
The PIAAC results are nationally representative and therefore reflect countries as they are: highly diverse or not. PIAAC collects extensive information about respondents' background and therefore support analyses that take into account differences in the amount of diversity across countries. The international PIAAC report produced by the OECD presents some analyses that examine issues of diversity.
As an international assessment of adult competencies, PIAAC differs from student assessments in several ways. PIAAC assesses a wide range of ages (16 through 65) whereas student assessments target a specific age (e.g., 15-year-olds in the case of PISA) or grade (e.g., grade 4 in PIRLS). PIAAC is a household assessment (i.e., an assessment administered in individuals' homes), whereas the international student assessments (PIRLS, PISA, and TIMSS) are conducted in schools. The skills that are measured in each assessment also differ based on the goals of the assessment. Both TIMSS and PIRLS are curriculum-based and are designed to assess what students have been taught in school in specific subjects, such as science, mathematics or reading, using multiple-choice and open-ended test questions. In contrast, PIAAC and PISA are "literacy" assessments, designed to measure performance in certain skill areas at a broader level than school curricula. So while TIMSS and PIRLS aim to assess particular academic knowledge that students are expected to be taught at particular grades, PISA and PIAAC encompass a broader set of skills that students and adults have acquired throughout life.
PIAAC has improved and expanded on the cognitive frameworks of previous large-scale literacy assessments, including NALS, NAAL, IALS, and ALL, and has added an assessment of problem solving via computer, which was not a component of these earlier surveys. In addition, PIAAC is capitalizing on prior experiences with large-scale assessments in its approach to survey design and sampling, measurement, data collection procedures, data processing, and weighting and estimation. Finally, the most significant difference between PIAAC and previous large-scale assessments is that PIAAC is administered on laptop computers, and is designed to be a computer-adaptive assessment so respondents will receive groups of items targeted to their performance levels (respondents not able to or not wishing to take the assessment were provided with an equivalent paper and pencil version of the literacy and numeracy items). Because of these differences, PIAAC introduces a new set of scales to measure adult literacy, numeracy, and problem solving. Some scales from these previous adult assessments have been mapped to the PIAAC scales so that performance can be measured over time.
PISA and PIAAC both emphasize knowledge and skills in the context of everyday situations, asking students and adults to perform tasks that involve real-world materials as much as possible. PISA is designed to show the knowledge and skills 15-year-old students have accumulated within and outside of school. It is intended to provide insight into what students who are about to complete compulsory education know and are able to do. PIAAC focuses on adults who are already eligible to be in the workforce, and aims to measure the set of literacy, numeracy, and technology-based problem solving skills an individual needs in order to function successfully in society. Therefore, PIAAC is not directly measuring the academic skills or knowledge adults may have learned in school. The PIAAC assessment focuses on tasks adults may encounter in their lives at home, work, or in their community.
PIAAC has improved and expanded on the cognitive frameworks of previous large-scale literacy assessments, including NAAL, IALS, and ALL, and has added an assessment of problem solving via computer, which was not a component of these earlier surveys. In addition, PIAAC is capitalizing on prior experiences with large-scale assessments in its approach to survey design and sampling, measurement, data collection procedures, data processing, and weighting and estimation. Finally, the most significant difference between PIAAC and previous large-scale assessments is that PIAAC is administered on laptop computers, and is designed to be a computer-adaptive assessment so respondents will receive groups of items targeted to their performance levels (respondents not able to or not wishing to take the assessment were provided with an equivalent paper-and-pencil version of the literacy and numeracy items). Because of these differences, PIAAC introduces a new set of scales to measure adult literacy, numeracy, and problem solving. Some scales from these previous adult assessments have been mapped to the PIAAC scales so that performance can be measured over time.
Trends are only reported for countries that participated in both IALS and ALL. Of the PIAAC countries, 6 countries participated in ALL and 14 participated in IALS. All 6 countries which participated in ALL also participated in IALS.
This has been a consistent trend in the United States across IALS, ALL, and PIAAC. It is probably related to a number of causes but we don't currently know why this is true and an answer can only be postulated after education researchers examine the rich data that was collected by PIAAC.
Each country can collect data for subgroups of the population that have national importance. In some countries, these subgroups are identified by language usage, in other countries they are distinguished by tribal affiliation. In the United States, different racial and ethnic subgroups are of national importance. However, categories of race and ethnicity are social and cultural categories that differ greatly across countries. As a result, they cannot be accurately compared across countries.
In the United States, PIAAC results can only be reported at the national level because of the size of the U.S. sample. In the United States, 5,000 adults participated in PIAAC, which are not enough respondents to produce accurate estimates at the state or county level. However, NCES is in the process of reviewing plans for producing state-level (synthetic) estimates.
PIAAC collects extensive information on educational attainment and years of schooling. For the purposes of cross-country comparison, the education level classifications of each country are standardized on what is known as the International Standard Classification of Education (ISCED). This classification allows for cross-country comparisons of educational attainment. For example, the ISCED level for short-cycle tertiary education (ISCED level 5) is equivalent to an Associate degree in the United States, and therefore comparisons of adults with Associate degrees or their equivalents can be made across countries using this classification. Please note that the education variables in PIAAC 2012 were classified using the ISCED97.
PIAAC assesses in the official language or languages of each participating country. Based on a 1988 congressional mandate and the 1991 National Literacy Act, the U.S. Department of Education is required to evaluate the status and progress of adults' literacy in English. However, in order to obtain background information from a wide range of respondents in the United States, the background questionnaire was administered in both English and Spanish.
The following 23 countries or regions participated in PIAAC 2012 and are included in the U.S. national report published by NCES: Australia, Austria, Belgium, Canada, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Ireland, Italy, Japan, Republic of Korea, Netherlands, Norway, Poland, Slovak Republic, Spain, Sweden, United Kingdom, and the United States.
The Russian Federation also participated in 2012 but their data are not included in NCES report or data explorer due to technical issues.