Progress in International Reading Literacy Study (PIRLS)



1. Overview

PROGRESS IN INTERNATIONAL READING AND LITERACY STUDY:
Three assessments:
  • PIRLS
  • ePIRLS
  • digitalPIRLS
Three aspects of reading literacy:
  • Purpose for reading
  • Processes of comprehension
  • Reading behaviors and attitudes
Six sets of questionnaires:
  • Student questionnaire
  • ePIRLS student questionnaire
  • Learning to read (home) survey
  • Teacher questionnaire
  • School principal questionnaire
  • Curriculum Questionnaire

The Progress in International Reading Literacy Study (PIRLS) is a large international comparative study of the reading literacy of fourth-grade students. The study is conducted by the International Association for the Evaluation of Educational Achievement (IEA), with national sponsors in each participating education system (previously referred to as education systems). The National Center for Education Statistics (NCES), in the Institute of Education Sciences at the U.S. Department of Education, is responsible for the implementation of PIRLS in the United States. Reading literacy is one of the most important abilities that students acquire as they progress through their early school years. It is the foundation for learning across all subjects, it can be used for recreation and for personal growth, and it equips young children with the ability to participate fully in their communities and the larger society. Participants in PIRLS include both countries and subnational entities, both of which are referred to as education systems. PIRLS focuses on the achievement and reading experiences of children in grades equivalent to fourth grade in the United States. The study includes a written test of reading comprehension and a series of questionnaires focusing on the factors associated with the development of reading literacy. PIRLS was administered in 2001 to students in 35 education systems, in 2006 to students in 45 education systems, in 2011 to students in 53 education systems, and in 2016 to students in 61 education systems.

Purpose

PIRLS is a carefully constructed reading assessment, consisting of a test of the reading literacy of fourth-grade students and questionnaires to collect information pertaining to fourth-grade students' reading literacy evaluation. PIRLS has four goals: (1) develop internationally valid instruments for measuring reading literacy suitable for establishing internationally comparable literacy levels in each of the participating education systems; (2) describe on one international scale the literacy profiles of fourth-graders in school in each of the participating education systems; (3) describe the reading habits of fourth-graders in each participating education system; and (4) identify the home, school, and societal factors associated with the literacy levels and reading habits of fourth-graders in school.

Components

PIRLS assesses four broad-based comprehension processes within each of the two purposes for reading: focus on and retrieve explicitly stated information; make straightforward inferences, interpret and integrate ideas and information; and evaluate and critique content and textual elements.

Also, PIRLS focuses on three aspects of reading literacy: purposes for reading; processes of comprehension; and student reading behaviors and attitudes. The first two aspects are measured through the PIRLS assessment component, which is administered to each participating student. The third dimension, reading behaviors and attitudes, is measured through a separate component of background questionnaires. In 2016, the PIRLS administration included the PIRLS assessment as well as ePIRLS, an assessment of online informational reading.

PIRLS 2021 will present a new digital web-based delivery system called digitalPIRLS. digitalPIRLS will be offered so countries can take advantage of a full computer-based assessment. The digitalPIRLS assessments will include the ePIRLS assessment of online reading initiated in 2016. With digitalPIRLS, countries will experience greater operational efficiency in translation and translation verification, data entry, and scoring, without the need for printing or shipping. digitalPIRLS will be offered as a web-based system via school-based or IEA web servers, or a USB drive connected locally to a PC with the Windows Operating System. As an alternative to digitalPIRLS, countries may administer PIRLS 2021 in paper format. ePIRLS is available in 2021 only in conjunction with digitalPIRLS. For more information on digitalPIRLS, please visit the IEA website at https://www.iea.nl/.

Source versions of all instruments (assessment booklets, the ePIRLS assessment, questionnaires, and manuals) were prepared in English and translated into the primary language or languages of instruction in each education system. In addition, it was sometimes necessary to adapt the instrument for cultural purposes, even in countries that use English as the primary language of instruction. All adaptations were reviewed and approved by the International Study Center to ensure they did not change the substance or intent of the question or answer choices.

The first aspect of the assessment component that is targeted by PIRLS is purposes of reading. The purposes of reading component encompasses the two main reasons why young students read printed materials: for literary experience and for the acquisition and use of information. To measure the ability of students to read for literary experience, fictional texts are used; to measure students' skills for acquiring and using information, nonfictional texts are used. In 2016, literary experience and acquiring and using information each made up 50 percent of this aspect of the PIRLS reading assessment. The second aspect of the PIRLS assessment component is processes of comprehension, which describes how young readers interpret and make sense of text. In 2016, this aspect was composed of four categories: focusing on and retrieving explicitly stated information (20 percent), making straightforward inferences (30 percent), interpreting and integrating ideas and information (30 percent) and evaluating and critiquing content and textual elements (20 percent).

Both PIRLS and PIRLS Literacy devote half of the assessment passages to each of the purposes for reading, while the ePIRLS online assessment focuses solely on reading to acquire and use information. The ePIRLS approach simulates websites from the Internet, through which students can navigate to accomplish school-based research projects or tasks. Because PIRLS Literacy is designed for students earlier in the process of learning to read, a larger percentage of items (50 percent of the assessment) is devoted to measuring foundational reading comprehension processes—the ability to focus on and retrieve explicitly stated information. Also, PIRLS Literacy has shorter reading passages with easier vocabulary and syntax.

The second component, background questionnaires, collects information on reading behaviors and attitudes (the third aspect of reading literacy targeted by PIRLS), and helps to provide a context for the performance scores. These questionnaires focus on such topics as students' attitudes and beliefs about learning, their habits and homework, and their lives both in and outside of school; teachers' attitudes and beliefs about teaching and learning, teaching assignments, class size and organization, instructional practices, and participation in professional development activities; and principals' viewpoints on policy and budget responsibilities, curriculum and instruction issues, and student behavior, as well as descriptions of the organization of schools and courses.

Assessment. In the main PIRLS assessment, each student completes a reading comprehension assessment booklet, which contains two blocks of passages that form the foundation of the PIRLS reading literacy test. Half of the assessment blocks contain passages of literary text which include realistic stories and traditional tales, and half contain passages of informational texts including chronological and non-chronological articles, a biographical article, and an informational leaflet. The passages of text are followed by questions about the text, which the student answers using constructed-response and multiple-choice response options.

The PIRLS assessment passages are submitted and reviewed by PIRLS countries and are geared to a fourth-grade level. To provide good coverage of each reading literacy domain, a total of eight blocks of test items were developed (for the 2001 assessment), representing over five hours of testing time. Each block of assessment material contained from 11 to 14 items that together represent at least 15 score points. The PIRLS reading test includes 98 items altogether, representing a total of 133 score points.

Student testing time was kept to 80 minutes per student: 40 minutes for each of the two blocks, plus an additional 15-30 minutes for the student questionnaire. Using a matrix sampling technique, the eight blocks of reading comprehension passages were used to create ten different student booklets that each contained two blocks of text. Possible booklet combinations included 1) two blocks of literary passages; 2) two blocks of informational text; or 3) one block of literary text and one block of informational text. The distribution of blocks across booklets "links" the booklets to enable the achievement data to be scaled using item response theory methods, and enables PIRLS to report results from a representative sample for more assessment items than can fit in one booklet without making the assessment longer for the student.

The PIRLS 2006 assessment was based on ten passages: five literary and five informational. Each passage was accompanied by approximately twelve questions, with the total assessment having 126 items. Two of the literary passages and two of the informational passages (so four blocks) were retained from the 2001 assessment in order to measure trends, with the rest of the passages (three literary and three informational) being new. These ten blocks or passages were distributed across thirteen booklets for PIRLS 2006, and included 126 items across the 10 assessment blocks comprising a total of 167 score points.

The PIRLS 2011 assessment contained ten reading passages: two from 2001 and 2006, four from 2006 only, and four new passages. The assessment consisted of 135 items, representing over six hours of testing time. The ten blocks were distributed across twelve assessment booklets for 2011. The use of the six common passages (or "trend blocks") from the 2001 and 2006 assessments allows for the analysis of changes in reading literacy over the ten-year period for the education systems that participated in both assessment administrations.

In 2016, both PIRLS and PIRLS Literacy consisted of 12 passages/blocks, each of which was expected to require 40 minutes of student testing time. Six of the ten PIRLS blocks were included in previous PIRLS assessments: two in all three assessments (2001, 2006, and 2011), two in both PIRLS 2006 and PIRLS 2011, and two in PIRLS 2011 only. Ten blocks were developed specifically for PIRLS Literacy; four of the passage and item blocks were previously used in 2011 as part of prePIRLS, and two blocks came from the main PIRLS assessment.

The complete ePIRLS assessment consists of five school-based online reading tasks, each of which involves 2–3 different websites totaling 5 to 10 web pages, together with a series of comprehension questions based on the task. Similar to the PIRLS and PIRLS Literacy passages, each task with accompanying questions takes 40 minutes to complete. In order to keep student response burden to a reasonable level, each individual student completes just two ePIRLS tasks, followed by 5 minutes for a short online questionnaire.

PIRLS 2021 will include 18 passages with various response items (18 booklets for both digital format or paper format).  Each booklet consists of two parts: one literary experience passage and one informational reading passage/items. PIRLS 2021 will employ a group-adaptive design. All countries will administer the same reading passages and items, but the rate at which different test forms are distributed in a country will be tailored to the population. This innovative adaptive design will improve PIRLS' measurement of reading at all levels of the distribution for countries with varying reading proficiency while also increasing student engagement. For more information on PIRLS 2021, please visit the IEA website at https://www.iea.nl/.

Top

Background Questionnaires. An important part of the PIRLS design is a set of questionnaires targeting factors related to reading literacy (i.e., reading behaviors and attitudes). To accomplish this, questionnaires are completed by principals, teachers, parents, and students. Prior to each new PIRLS data collection, previous versions of the questionnaires are reviewed extensively by the participating countries' National Research Coordinators (NRCs), as well as by the Questionnaire Development Group, known as Questionnaire Item Review Committee (QIRC). The QIRC comprises 10–12 experienced NRCs from different participating countries who have analyzed PIRLS data and are using the data in their countries. Like the assessment items, all questionnaire items were field tested and revised if necessary prior to their inclusion in the final questionnaires.

PIRLS included four sets of background questionnaires in 2001: one each for the tested student, the parent or primary care giver of the tested student, the student's reading teacher, and the principal of the school to which the tested student attended. PIRLS 2006 also administered questionnaires to students, parents, teachers, and school principals. In addition, the NRC in each country completed a new, online curriculum questionnaire that provided data on the country's goals for reading instruction. For 2011, the background questionnaires for teachers, students, and principals were revised somewhat from previous years, and online versions of the school and teacher questionnaires were offered to respondents as the primary mode of data collection. The curriculum questionnaire was also administered. In 2016, the questionnaires were revised from 2011, and the ePIRLS student questionnaire was added.

Student questionnaire. Each student taking the PIRLS reading assessment completes the student questionnaire. The questionnaire asks about students' home environment, such as languages spoken at home, books in the home, and other home resources for learning. This questionnaire also gathers information on student experiences in school, including feelings of school belonging and whether they are victims of bullying. Finally, the student questionnaire gathers data on out-of-school reading habits and attitudes toward reading, including whether they like reading, their confidence in reading, and their engagement in reading lessons. The student questionnaire requires 15–30 minutes to complete.

ePIRLS student questionnaire. Students also participating in ePIRLS complete a brief questionnaire as part of this computer-based assessment. The questionnaire asks students about their level of competency and experience using computers and finding information on the Internet. This questionnaire requires 5 minutes to complete.

Teacher questionnaire. The reading teacher of each fourth-grade class sampled for PIRLS also completes a questionnaire. This questionnaire asks teachers about classroom contexts for reading instruction, such as characteristics of the class, reading instructional time, and instructional approaches. The questionnaire also asks about teacher characteristics, such as their career satisfaction, education, and recent professional development activities. This questionnaire requires about 35 minutes to complete.

School questionnaire. The principal of each school that is sampled is asked to respond to the school questionnaire. The principals are asked about school characteristics, such as student demographics, the school environment, and the availability of school resources and technology. The questionnaire also includes items focusing on the principal's leadership role, education, and experience. It is designed to take about 30 minutes.

Curriculum questionnaire. First used in PIRLS 2006, this questionnaire is administered to NRCs. They are asked to provide information about their national policies on reading curricula, goals and standards for reading instruction, and time specified for reading instruction, as well as information on preprimary education and teacher education policies.

Home questionnaire. The home questionnaire, entitled the Learning to Read Survey, is addressed to the parents or primary caregivers of each student taking part in the PIRLS 2016 data collection. This short questionnaire solicits information on the home context, such as languages spoken in the home, parents' reading activities and attitudes toward reading, and parents' education and occupation. The questionnaire also collects data on the students' educational activities and experiences outside of school including early childhood education, early literacy and numeracy activities, and the child's reading readiness at the beginning of primary school. This questionnaire is designed to take 10–15 minutes to complete. The home questionnaire is not administered in the United States.

Periodicity

PIRLS is administered once every 5 years, near the end of the school year in each education system. PIRLS was conducted in 2001, 2006, 2011, and 2016. The next administration is scheduled for 2021.

Data Availability

Information on the availability of data for PIRLS can be found at https://nces.ed.gov/surveys/pirls/datafiles.asp.