What PIAAC Measures

The Definition of Literacy Domain expanded on the definition of literacy used in the International Adult Literacy Survey (IALS) and the Adult Literacy and Lifeskills Survey (ALL)¹ and provides a broad definition of literacy:

"Literacy is understanding, evaluating, using and engaging with written text to participate in society, to achieve one's goals, and to develop one's knowledge and potential."

This definition (a) highlights the ranges of cognitive processes involved in literacy; (b) focuses on a more active, participatory role of individuals in society; and (c) includes a range of text types, such as narrative and interactive texts, in both print and electronic formats.

While this is broader than the definition used in IALS and ALL, selected items from those assessments are used to provide a link to IALS and ALL. PIAAC items include continuous texts (e.g., sentences and paragraphs), noncontinuous texts (e.g., schedules, graphs, maps), and electronic texts (including hypertext, or text in interactive environments, such as forms and blogs). Task activities are presented in home, work, and community contexts, addressing the various purposes that adults pursue in their lives.

Based on the PIAAC framework, literacy tasks include items (in both computer-based and paper-and-pencil modes) that cover a range of difficulties—low, middle, and high—to present a comprehensive picture of the range of skills of adults in each country.

¹ IALS and ALL definition: Literacy is using printed and written information to function in society to achieve one's goals and to develop one's knowledge and potential.

For PIAAC, literacy was defined as follows:

"Literacy is understanding, evaluating, using and engaging with written texts to participate in society, to achieve one’s goals, and to develop one’s knowledge and potential."

Some key terms within this definition are explained below.

"Understanding"

A basic task for the reader is constructing meaning, both large and small, literal and implicit, from text. This can be as basic as understanding the meaning of the words to as complex as comprehending the underlying theme of a lengthy argument or narrative. Certainly, evaluating or using a text implies some level of understanding and so provides an indirect measure of it, but it is the intent of the PIAAC assessment to have a more direct measure. The components framework provides the construct to support basic understanding, but the literacy assessment itself should also include tasks that explicitly tap more complex understanding, such as the relationships between different parts of the text, the gist of the text as a whole, and insight into the author’s intent. Readers also have to understand the social function of each text and the way this influences structure and content.

"Evaluating"

Readers continually make judgments about a text they are approaching. They need to assess whether the text is appropriate for the task at hand, determining whether it will provide the information they need. They have to make judgments about the truthfulness and reliability of the content. They need to account for any biases they find in the text. And, for some texts, they must make judgments about the quality of the text, both as a craft object and as a tool for acquiring information.

Such judgments are especially important for electronic texts. While published print information carries a sense of legitimacy, especially where the reader can assume there has been some review and edit process, sources for online information are more varied, ranging from authoritative to postings with unknown or uncertain authenticity. All information must be evaluated in terms of accuracy, reliability, and timeliness, but this is particularly important with online material.

"Using"

Much adult reading is directed toward applying the information and ideas in a text to an immediate task or to reinforce or change beliefs. Nearly all the tasks in previous international assessments have been of this kind. In some cases, using a text in this way requires just minimal understanding, getting the meaning of the words with some elementary recognition of structure (many menus, for example). In others, it requires using both syntactic and more complex structural understanding to extract the information. In all cases though, the reader approaches the text with a specific task in mind.

"Engaging with"

Many adults appear to read text only when some task requires them to do so. Others sometimes also read for the pleasure it brings them. That is, adults differ in how they engage with text and how much of a role reading plays in their lives. Studies have found that engagement with (that is, the attitude toward and practice of) reading is an important correlate with the direct cognitive measures. As such, it is necessary to understand these differences to get a full picture of adult literacy.

"Written text"

Previous literacy assessments have focused primarily on informative texts of both continuous and noncontinuous form. It is the intention of the new construct to expand the range of texts to include a greater variety of text types, such as narrative and interactive texts, and a greater variety of media. Until recently, most adult reading was of material printed on paper. Now, adults need to access and use text that is displayed on a screen of some kind, whether computer, a mobile device, or an ATM, or a Blackberry or iPhone. The PIAAC definition encompasses all these.

It is worth noting that including electronic text opens the assessment to new types of text and content. While one can find examples of similar texts in paper, they are much less common in that form. Some of these novel form/content combinations include interactive texts, such as exchanges in comments sections of blogs or in e-mail response threads; multiple texts, whether displayed at the same time on a screen or linked through hypertext; and expandable texts, where a summary can be linked to more detailed information if the user chooses.

"Participate in society"

While earlier definitions referred to the role of literacy in “functioning” in society, the PIAAC use of “participating” is meant to focus on a more active role for the individual. Adults use text as a way to engage with their social surroundings, to learn about and to actively contribute to life in their community, close to home and more broadly. And for many adults, literacy is essential to their participation in the labor force. In this, we recognize the social aspect of literacy, seeing it as part of the interactions between and among individuals.

"Achieve one’s goals"

Adults have a range of needs they must address, from basic survival to personal satisfaction and to professional and career development. Literacy is increasingly complicit in meeting those needs, whether it is to simply find one’s way through shopping or to negotiate complex bureaucracies, whose rules are commonly available only in written texts. It is also important in meeting adult needs for sociability, for entertainment, and leisure and for work.

"Develop one’s potential"

Surveys suggest that many adults engage in some kind of learning throughout their life, much of it self-directed and informal. Much of this learning requires some use of text, and, as individuals want to improve their life, whether at work or outside, they need to understand, use, and engage with printed and electronic materials.

For the complete PIAAC literacy framework, see

PIAAC Literacy: A Conceptual Framework

http://www.oecd-ilibrary.org/education/piaac-literacy-a-conceptual-framework_220348414075

The primary goal of PIAAC’s numeracy assessment is to evaluate basic mathematical and computational skills that are considered fundamental for functioning in everyday work and social life. In the Definition of Numeracy Domain, numeracy is defined as:

"the ability to access, use, interpret, and communicate mathematical information and ideas, to engage in and manage mathematical demands of a range of situations in adult life"

This definition (a) highlights the ranges of cognitive processes involved in numeracy; (b) focuses on a more active, engaged role of individuals in society; and (c) includes a range of representations of mathematical information, such as text or graphs.

This definition is compatible with that used in the Adult Literacy and Lifeskills Survey (ALL),¹ and selected items from that assessment are used to provide a link to ALL. PIAAC items are represented in multiple ways, including pictures, mathematical symbols, formulas, diagrams, maps, graphs, tables, texts, and technology-based displays. Task activities are presented in home, work, and community contexts, addressing the various purposes that adults pursue in their lives.

Based on the PIAAC framework, numeracy tasks include items (in both computer-based and paper-and-pencil modes) that cover a range of difficulties—low, middle, and high—to present a comprehensive picture of the range of skills of adults in each country.

¹ALL definition: Numeracy is the knowledge and skills required to effectively manage and respond to the mathematical demands of diverse situations.

For PIAAC, numeracy was defined as follows:

"Numeracy is the ability to access, use, interpret, and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life."

This definition captures essential elements in numerous conceptualizations of numeracy in the extant literature; it is compatible with the definition used in the Adult Literacy and Life Skills Survey (ALL), an international assessment of adult skills conducted in 2003 and 2008, and appears to provide a solid basis from which to develop an assessment scale for PIAAC with its emphasis on competencies in the information age. The inclusion of "engage" in the definition signals that not only cognitive skills but also dispositional elements—i.e., beliefs and attitudes—are necessary for effective and active coping with numeracy situations. It is also important to note that while the definition of numeracy for PIAAC has been developed in the context of an assessment program, it has been crafted so as to contribute to public dialogue regarding the goal of educational and social interventions focused on developing adult competencies in general and adult numeracy and related mathematical and statistical skills and dispositions in particular.

However, since numeracy is a broad, multifaceted construct referring to a complex competency, the definition of numeracy given above should not be considered by itself but should be coupled with a more detailed definition of numerate behavior and with further specification of the facets of numerate behavior. This pairing is essential to enable operationalization of the construct of numeracy in an actual assessment, thereby contributing to the assessment’s validity and interpretability and further broadening the understanding of key terms appearing in the definition itself. Consequently, a definition of numerate behavior similar in general terms to the one used for the ALL survey, but shorter, has been adopted for PIAAC:

"Numerate Behavior involves managing a situation or solving a problem in a real context, by responding to mathematical content/information/ideas represented in multiple ways..."

Table 1: Numerate behavior: Key facets and their components

The definition of numerate behavior pertains to four facets: Contexts, Responses, Mathematical Content/Information/Ideas, and Representations. Table 1 lists the components of the four facets, and these components are explained in more detail the next section. Table 1 is based on the original description of the facets of numerate behavior developed for the ALL survey, but some changes have been implemented, such as the addition of "access" and "evaluate/analyze" as possible responses, the merging of the content categories of "change" and "pattern and relationship," and the reference to "technology-based displays" as another representation mode.

It should be noted that the bottom section of Table 1 lists several enabling factors and processes whose activation also underlies numerate behavior. Most of these enabling factors and processes appeared in the ALL conceptual framework, but some changes were introduced for PIAAC, such as the inclusion of "adaptive reasoning and mathematical problem-solving" as a separate factor. Overall, the definition of numerate behavior presented earlier, together with the detail in table 1 and the further explanations within the complete numeracy framework (see below), provided a roadmap for the development of a numeracy scale for PIAAC.

For a more complete description of the numeracy framework, see:

PIAAC Numeracy: A Conceptual Framework

http://www.oecd-ilibrary.org/education/piaac-numeracy-a-conceptual-framework_220337421165

Problem solving in technology-rich environments (PS-TRE), also known as digital problem solving, is a type of digital problem solving that was introduced in PIAAC’s Cycle I as an innovative addition to both adult literacy and international large-scale assessments. In the Definition of Digital Problem-Solving Domain, PS-TRE is defined as:

"using digital technology, communication tools, and networks to acquire and evaluate information, communicate with others, and perform practical tasks."

This definition focuses on the skills and abilities required for (a) accessing and using digital environment tools and resources, (b) evaluating the validity of the information, and (c) communicating information.

PS-TRE assesses the cognitive processes of problem solving: goal setting, planning, selecting, evaluating, organizing, and communicating results. The environment in which PS-TRE assesses these processes was meant to reflect the reality that digital technology has revolutionized access to information and communication capabilities over the past decades. In particular, the Internet has immensely increased instantaneous access to large amounts of information in multiple formats and expanded the capabilities of instant voice, text, visual, and graphic communications across the globe.

In order to effectively operate in the digital environment, it is necessary to have mastery of information and communications technology (ICT) skills, including (a) skills associated with manipulating input and output devices (e.g., the mouse, keyboard, and digital displays), (b) awareness of concepts and knowledge of how the environment is structured (e.g., scrollbars, hyperlinks, files, folders, and different types of menus or buttons), and (c) the ability to interact effectively with digital information (e.g., how to open and close apps, move or highlight text, and save, delete, and send files). Such interaction involves familiarity with electronic apps, files, texts, images, graphics and numerical data, as well as the ability to locate, evaluate, and critically judge the validity, accuracy, and appropriateness of accessed information. These skills constituted the core aspects of the PS-TRE assessment.

PS-TRE tasks were performed in simulated software applications using commands and functions found in the common technology environments at the time—browser-based e-mail, web pages, and spreadsheets. These tasks ranged from purchasing particular goods or services online and finding interactive health information to managing personal information and business finances.

Based on the PIAAC framework, PS-TRE tasks include items (only in the computer-based mode and not in the paper-and-pencil mode) that cover a range of difficulties—low, middle, and high—to present a comprehensive picture of the range of skills of adults in each country.

In the context of the PIAAC survey, problem solving in technology-rich environments is defined as follows:

"Problem solving in technology-rich environments involves using digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks. The first PIAAC problem-solving survey will focus on the abilities to solve problems for personal, work and civic purposes by setting up appropriate goals and plans, accessing and making use of information through computers and computer networks."

The two sentences in the definition each serve a specific purpose. The first sentence is aimed at providing a broad basis for the first as well as subsequent surveys of PS-TRE. The second sentence acknowledges some constraints that limit the scope of the first survey. We provide below a series of more specific comments on the words and phrases used in this definition.

"using digital technology, communication tools and networks"

PIAAC focuses on problems that are specifically related to the use of ICT. The problem-solving context means that routine or basic ICT skills will not be central to the framework. Instead, PS-TRE will focus on situations that involve the active construction of goals and strategies on the part of the user. We also acknowledge the increasing diversity and versatility of digital technologies, and we emphasize that a proper assessment of PS-TRE should not be limited to traditional desktop computing. Instead, we envision that mobile and integrated technologies may be involved in new types of problem solving that will need to be represented in future assessments.

"to acquire and evaluate information"

This phrase acknowledges that most uses of digital technologies involve the use of symbolic information, such as texts, graphics, links, and commands. Symbolic information is used as part of human computer interfaces (e.g., icons, commands) and it constitutes the primary content of most computer applications (e.g., word processor, spreadsheet, Internet browser, and e-mail applications). The phrase also emphasizes that computers and computers networks, such as the Internet, mostly offer a multiplicity of information sources from which the relevant and reliable pieces have to be chosen for their respective purposes.

"communicate with others"

An important role of digital technologies is to provide powerful and flexible means for people to communicate with each other. Examples include e-mail, chats, short message systems, and IP audiovisual communication. Digital communication may take place in the context of purposeful, problem-like situations and therefore it is an integral part of the PIAAC PS-TRE construct.

"and perform practical tasks"

The ability to solve problems with digital technologies is tightly related to the achievement of personal, civic and work-related purposes, which, in turn, take the form of concrete, practical tasks. Examples include shopping, learning about laws and regulations, and organizing teamwork through online agendas and reservation systems. The problems assessed in PIAAC will use authentic, meaningful scenarios based on surveys of computer uses and input from participating countries.

"The first PIAAC problem-solving survey"

This is the first attempt to assess PS-TRE on a large scale and as a single dimension. This creates many challenges concerning the definition of tasks and the practical collection of data. Furthermore, digital technologies keep evolving at a rapid pace, as do the personal, social, and work-related uses of those technologies. While setting the stage for further rounds of surveys, the present framework will take a perspective on PS-TRE that takes into consideration feasibility issues as well as possible evolutions of technology and technology uses.

"will focus on the abilities to solve problems for personal, work and civic purposes"

In order to reflect the pervasiveness of ICT in society, PIAAC PS-TRE will assess problem-solving ability based on scenarios that pertain to these three important contexts.

"by setting up appropriate goals and plans,"

An assessment of problem-solving capacity should focus on situations where test takers cannot immediately reach their goal based on routine, mechanistic sets of actions. Instead, we focus on tasks that require test takers to actively construct a solution based on the resources available in the assessment environment.

"accessing and making use of information"

Again, this phrase emphasizes a specific aspect of PS-TRE, namely, that these are often information-rich problems that require individuals to access, interpret and integrate multiple sources of information.

"through computers and computer networks."

There is more to "technology-rich environments" than merely personal computers. A full assessment of PS in TRE would require a range of devices that mimic the diversity and versatility of the digital technologies of today's world. However, for feasibility reasons, this first survey will be limited to problems requiring the use of computers and internet-based services.

For a more complete description of the problem-solving framework, see:

PIAAC Problem Solving in Technology-Rich Environments: A Conceptual Framework

http://www.oecd-ilibrary.org/education/piaac-problem-solving-in-technology-rich-environments-a-conceptual-framework_220262483674

The reading components assessment framework builds upon the basic principle that comprehension—i.e., the “meaning construction” processes of reading—is built upon a foundation of knowledge of how one’s language is represented in one’s writing system (i.e., the component print skills). This basic principle of learning to read has now been widely researched and accepted internationally (Curtis, 1980; Oakhill, Cain, and Bryant, 2003; Perfetti, 1985, 2003; Sabatini, 2003; Strucker, Yamamoto, and Kirsch, 2004). Evidence of an individual’s level of print skill can be captured in tasks that examine a reader’s ability and efficiency in processing the elements of the written language: letters/characters, words, sentences, and larger, continuous text segments.

Another key principle guiding the framework is that as one becomes proficient in reading, the component skills will become more efficient and integrated. As learners, we spend extra time, effort, and energy to solve problems that are novel. On familiar tasks, we can often respond accurately, quickly, with seemingly little conscious effort. When the tasks are easy, we can spend more effort solving and learning from more complex problems and tasks. This is a basic tenet of “automaticity” (LaBerge and Samuels, 1974) and verbal efficiency theory (Perfetti, 1985, 1992, 2003). Component efficiency is typically indexed by assessing the speed or rate of processing, as well as accuracy. Speed or rate can be approximated by recording the time it takes to complete certain tasks or by setting a time limit and observing how many items are completed in the time frame allotted.

Item Design

Word Meaning (Print Vocabulary)

In the reading component skills framework, we seek to determine whether individuals can identify in print those words that are in the everyday listening lexicon of average adult speakers of the language: that is, the emphasis is on the everyday words of the language. This would be the language used in the neighborhood or market. It would be the language of popular media such as newspaper, radio, and television. This is the most cross-country, comparable vocabulary.

The Word Meaning (Print Vocabulary) measure presents an image and four word choices. The respondent must select the word choice that matches the picture. Target words are concrete, imageable nouns of common objects. Distractors are designed to tap similar semantic and/or orthographic features of the target word.

Sentence Processing

A variety of psychological studies of reading show that the sentence is a natural breakpoint in the reading of continuous text (e.g., Kintsch, 1998). A skilled reader will generally pause at the end of each sentence. A variety of operations are typically performed, including encoding the propositions of the sentence, making anaphoric inferences, relating meaning units to background knowledge and to previous memory of the passage as it unfolds, and deciding which meaning elements to hold in working memory.

The Sentence Processing measure presents sentences of increasing difficulty (as indexed by length) and asks the respondent to make a sensibility judgment about the sentence with respect to general knowledge about the world or about the internal logic of the sentence.

Passage Comprehension

In recent research, a silent reading assessment task design has gained empirical support as an indicator of basic reading fluency and comprehension. The design uses a forced-choice cloze paradigm: that is, a choice is given between a word that correctly completes a sentence in a passage and an option that is incorrect. The incorrect item is meant to be obviously wrong to a reader with some basic comprehension skills. The integration of decoding, word recognition, vocabulary, and sentence processing is required to construct the basic meaning of a short passage. Fluent, efficient performance on such a basic, integrated reading task is a building block for handling longer, more complex literacy texts and tasks.

The Passage Comprehension measure presents three passages, each with embedded cloze items. Passages were constructed based on the kinds of text types that adults typically encounter: narrative, persuasive, and expository.

For the complete Reading Components framework see:

PIAAC Reading Components: A Conceptual Framework

http://www.oecd-ilibrary.org/education/piaac-reading-component-a-conceptual-framework_220367414132