View Quarterly by:
This Issue | Volume and Issue | Topics
|
|||
| |||
This article was excerpted from the Statistical Analysis Report of the same name. The sample survey data are primarily from the Progress in International Reading Literacy Study (PIRLS). | |||
| |||
The Progress in International Reading Literacy Study of 2001 (PIRLS 2001) is an assessment of reading comprehension conducted by the International Association for the Evaluation of Educational Achievement (IEA). Thirty-five countries assessed the reading literacy of students in the upper of the two grades with the most 9-year-olds (fourth grade in most countries, including the United States). PIRLS 2001 provides comparative information on the reading literacy of these fourth-graders and also examines factors that may be associated with the acquisition of reading literacy in young children. PIRLS 2001 will help educators and policymakers by answering questions such as the following:
As the sponsor for PIRLS 2001 in the United States, the National Center for Education Statistics (NCES) is reporting findings from the study that compare the United States with other countries and that take a closer look at performance within the United States. The full report on the international study is available at www.pirls.org. Also available at this site is the PIRLS 2001 Technical Report (Martin, Mullis, and Kennedy 2003), which examines specific technical issues related to the assessment. Supporting data for the tables and analyses in this report are available at nces.ed.gov/surveys/pirls.
Background PIRLS 2001 follows by 10 years a prior IEA study of reading literacy called the IEA International Reading Literacy Study of 1991. Over the 10 years between these studies, progress has been made in the ways in which students are assessed and in the construction of the assessment instruments themselves. There has also been a shift in the design of the assessments. Thus, while PIRLS 2001 can trace its evolution from the 1991 IEA study, it is nevertheless a different study. PIRLS 2001 is the first in a planned 5-year cycle of international trend studies in reading literacy by the IEA. PIRLS is designed to assist participating countries in monitoring the reading literacy of their fourth-grade populations in comparison to other countries.
Construction and administration A group of distinguished international reading scholars, the Reading Development Group, was formed to construct the PIRLS 2001 Framework (see Campbell et al. 2001) and endorse the final reading assessment. Each country followed internationally prescribed procedures to ensure valid translations and representative samples of students. Quality Control Monitors were then appointed in each country to monitor the testing sessions at the schools to ensure that the high standards of the PIRLS 2001 data collection process were met. Reading literacy achievement was measured by using a selection of four literary passages drawn from children's storybooks and four informational texts. Submitted and reviewed by the PIRLS 2001 countries, the literary passages included realistic stories and traditional tales. The informational texts included chronological and nonchronological articles, a biographical article, and an informational leaflet.
Data collection Data were collected in the final months of the 2000–01 school year. In the United States, data were collected in the spring of 2001 from both public and private schools.
Definition and aspects of reading literacy PIRLS 2001 measures reading abilities at a time in students' schooling when most have learned how to read and are now using reading to learn. PIRLS 2001 defines reading literacy as follows: The ability to understand and use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers, and for enjoyment (Campbell et al. 2001, p. 3).In PIRLS 2001, three aspects of reading literacy are assessed: purposes of reading, processes of comprehension, and reading behavior and attitudes. The first two aspects of reading literacy form the basis of the written test of reading comprehension, while the student background questionnaire addresses the third aspect. Purposes of reading refers to the two types of reading that account for most of the reading young students do, both in and out of school: (1) reading for literary experience, and (2) reading to acquire and use information. In the assessment, narrative fiction is used to assess students' ability to read for literary experience, while a variety of informational texts are used to assess students' ability to acquire and use information while reading. The PIRLS 2001 assessment contains an equal proportion of text assessing each purpose. Processes of comprehension refers to ways in which readers construct meaning from the text. Readers (1) focus on and retrieve specific ideas, (2) make inferences, (3) interpret and integrate ideas and information, and (4) examine or evaluate text features. As shown in figure A, each process is assessed within each purpose of reading.
Figure A. Percentage of PIRLS assessment items devoted to reading purposes and processes NOTE: Detail may not sum to totals due to rounding. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 2 on p. 3 of the complete report from which this article is excerpted.)
PIRLS 2001 scores are reported on a scale of 0 to 1000, with an international average of 500 and a standard deviation of 100.1 For the 35 countries that participated in PIRLS 2001, figure B presents the average scores for three scales: the combined reading literacy scale and its two components, the literary and informational subscales.2 The average scores of U.S. students are compared to the average scores of students in other participating countries and the international average score.3
U.S. student performance on the combined reading literacy scale
U.S. student performance on subscales
1Met guidelines for sample participation rates only after replacement schools were included. 2National Defined Population covers less than 95 percent of National Desired Population. 3National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent. 4Canada is represented by the provinces of Ontario and Quebec (O, Q) only. 5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China. 6National Defined Population covers less than 80 percent of National Desired Population. 7Nearly satisfied guidelines for sample participation rates after replacement schools were included. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 3 on p. 5 of the complete report from which this article is excerpted.)
The average scores for reading literacy describe how a country performs overall compared to other nations, but they provide no information about the way scores are distributed within the countries. One country with an average score similar to another could have large numbers of high- and low-scoring students, while the other country could have large numbers of students performing at about the average score. Figure C details how scores are distributed across countries.
Average scores in figure B indicate how well the United States performs relative to other countries, but the scores do not indicate the proficiency required to reach a particular score. To gain a better understanding of what scores represent in terms of reading proficiency, PIRLS 2001 selected four cutoff points on the combined reading literacy scale labeled international benchmarks. These benchmarks were selected to correspond to the score points at or above which the lower quarter, median, upper quarter, and top 10 percent of fourth-graders in the international PIRLS 2001 sample performed.4 Student responses at the four benchmarks were analyzed to describe a set of reading skills and strategies displayed by fourth-graders at those points. These descriptions, together with the cut point scores, are listed in figure D.5
1Met guidelines for sample participation rates only after replacement schools were included. 2National Defined Population covers less than 95 percent of National Desired Population. 3National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent. 4Canada is represented by the provinces of Ontario and Quebec (O, Q) only. 5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China. 6National Defined Population covers less than 80 percent of National Desired Population. 7Nearly satisfied guidelines for sample participation rates after replacement schools were included. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 4 on p. 7 of the complete report from which this article is excerpted.) 1The responses of students who score within 5 points of each of the cut point scores were evaluated to determine reading skills and strategies displayed by fourth-graders at those points. Procedures used for anchoring these items to the benchmarks are explained more fully in the PIRLS Technical Report at www.pirls.org. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 5 on p. 8 of the complete report from which this article is excerpted.)
Achievement by sex In the United States and many other countries, policymakers and educators are interested not only in overall achievement but also in achievement by specific groups of students. For example, patterns of differences between boys and girls in reading achievement across countries can point to areas where additional educational resources might be focused.
U.S. achievement by race/ethnicity Another area of interest among policymakers and educators is the achievement of racial/ethnic groups. A number of countries that participated in PIRLS 2001 have large and diverse racial/ethnic groups. However, since these groups vary considerably across countries, it is not possible to compare their performance internationally. Thus, the findings in this section refer only to PIRLS 2001 results for the United States.
U.S. achievement by control of school On average, fourth-grade students in private schools in the United States score significantly higher than fourth-grade students in public schools on the combined reading literacy scale, and also on the literary and informational subscales. For example, on the combined reading literacy scale and the informational subscale, on average, fourth-grade students in private schools score 42 points higher than students in public schools. On the literary subscale, private school fourth-graders score an average of 45 points higher than public school fourth-graders.
U.S. achievement by poverty level in public schools One measure of poverty in U.S. public elementary schools is the percentage of students eligible for free or reduced-price lunch.7 In order to examine how fourth-graders' scores on the combined reading literacy scale are associated with their schools' poverty level (percentage of students receiving free or reduced-price lunch), U.S. public schools were classified into five groups: (1) schools with the lowest poverty levels of less than 10 percent; (2) schools with poverty levels ranging from 10 to 24.9 percent; (3) schools with poverty levels ranging from 25 to 49.9 percent; (4) schools with poverty levels ranging from 50 to 74.9 percent; and (5) schools with the highest poverty levels of 75 percent or more.8
1National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent. 2Canada is represented by the provinces of Ontario and Quebec (O, Q) only. 3Met guidelines for sample participation rates only after replacement schools were included. 4National Defined Population covers less than 95 percent of National Desired Population. 5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China. 6National Defined Population covers less than 80 percent of National Desired Population. 7Nearly satisfied national guidelines for sample participation rates after replacement schools were included. NOTE: All average score differences reported are statistically significant. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 7 on p.11 of the complete report from which this article is excerpted.) NOTE: Black includes African American, and Hispanic includes Latino. Race categories exclude Hispanic origin unless specified. The United States met guidelines for sample participation rates after replacement schools were included. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 9 on p. 13 of the complete report from which this article is excerpted.)
Reading curriculum and instructional time Do school principals and teachers encourage reading instruction through a variety of initiatives? What proportion of the school day is spent in reading instruction? Answers to these questions can give an indication of the emphasis that reading instruction receives in the curriculum of a country.
Teacher preparation and experience Examining teachers' preparation and tenure indicates the experience of teachers in the classroom. On the teacher questionnaire in PIRLS 2001, teachers were asked about the training they have received and the number of years they have been teaching.
1Significant difference between U.S. average and international average in this category. NOTE: The United States met guidelines for sample participation rates after replacement schools were included. SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 11 on p.16 of the complete report from which this article is excerpted.)
Reading outside of school for enjoyment To investigate the reading habits of fourth-graders outside of school, PIRLS asked students a series of questions about whether they read for fun outside of school and how often they did so. Students could indicate that they read for fun "every day or almost every day," "once or twice a week," "once or twice a month," or "never or almost never."
Choice of activities outside of school To learn more about students' reading habits, PIRLS 2001 asked students about their choice of reading materials and how often they read different types of texts when they are not in school.
PIRLS 2001 also asked students about their TV- and video-watching habits.
The sample items presented here show actual student responses and compare U.S. fourth-graders' performance to the international average. The items also demonstrate acceptable performance at the four benchmarks (top 10 percent, upper quarter, median, and lower quarter). The reading passage (exhibit A) and all of these items have been released to the public by IEA.
Exhibit A. One of the reading passages used in PIRLS 2001 SOURCE: Previously published on p. 20 of the complete report from which this article is excerpted.
Reading performance over time Ten years before PIRLS 2001 was administered, the IEA conducted the IEA International Reading Literacy Study of 1991. This study, like PIRLS 2001, assessed the reading literacy of fourth-graders in over 30 countries using 42 items taken from 6 reading passages. However, when a follow-up for the 1991 study was being planned, the IEA decided to discontinue it and develop a new assessment incorporating the latest approaches to measuring reading literacy (Campbell et al. 2001). This new study would become PIRLS 2001. In anticipation of the simultaneous release of PIRLS 2001 and the IEA International Reading Literacy Study of 1991, NCES commissioned a comparative analysis of the two assessments. Frameworks, passages, and items in both studies were reviewed and compared. Results indicate that the two studies are quite different. To cite a few examples: Reading passages in PIRLS 2001 were found to be "longer, more engaging, and more complex in most cases" than those found in the IEA International Reading Literacy Study of 1991 (Kapinus 2003, p. 8). PIRLS 2001 also used many more constructed-response (essay-type) questions and presented them in a way "that might have improved students' motivation to read and respond to the texts" (Kapinus 2003, p. 8). The analysis also found that, in general, PIRLS 2001 tapped skills "requiring deeper thinking" than those in the IEA International Reading Literacy Study of 1991 (Kapinus 2003, p. 8). Because of these and other differences, it is impossible to directly compare results from these two assessments. However, separately, each study provides important clues about how well students in these countries, including U.S. fourth-graders, perform in reading literacy. While participating in PIRLS 2001, some countries expressed interest in comparing reading performance between 1991 and 2001. Since comparisons between the two assessments were impossible, the IEA gave participating countries an opportunity to readminister the 1991 study during the PIRLS 2001 administration. This readministered study was identical in content, timing, and directions to that given to students in 1991 and allowed comparisons of the performance of students in 2001 with those in 1991. A separate sample of students was drawn in each country so as not to overburden students assessed in PIRLS 2001. Nine countries, including the United States, participated in the 2001 readministration of the IEA International Reading Literacy Study of 1991.
Performance on the IEA International Reading Literacy Study of 1991
Footnotes
1 The international average is the mean of all countries participating in the study calculated so that all participating countries have the same contribution to the average. The PIRLS 2001 scale average for each scale (the combined reading literacy scale and the literary and informational subscales) across countries was set to 500 and the standard deviation to 100. 2 Average scores for each country are based on a sample of students, rather than all students, and are estimates of the population value of all 9-year-olds in each country. The combined literacy scale is based on the distribution of scores on all the test items, while the subscales are based on only the items that belong to each subscale. Hence, the combined reading literacy score is not the statistical average of the scores of the two subscales. 3 No statistical adjustments (such as Bonferonni) are made while carrying out multiple comparisons between the United States and other countries. In order to be consistent with the comparisons carried out for the 4 Benchmarking in PIRLS describes the performance of students at four international benchmarks based on the distribution of scores and the pattern of items answered correctly. Proficiency levels for the National Assessment of Educational Progress (NAEP) (i.e., Basic, Proficient, and Advanced ) are established by the National Assessment Governing Board based on recommendations from broadly representative panels of educators and the general public who determine what students should know and be able to do at the three levels of performance in each subject area and in each grade assessed. 5 If students' reading achievement was distributed in the same way in every country, then each country would be expected to have approximately 10 percent of fourth-graders reaching the top 10 percent benchmark, 25 percent the upper quarter benchmark, 50 percent the median benchmark, and 75 percent the lower quarter benchmark. 6 Differences in scores by sex are not shown here for Kuwait due to low response rates on the question related to sex. However, the international average includes Kuwait's average scale score. 7 Data for the percentage of students eligible for free or reduced-price lunch in U.S. public elementary schools participating in PIRLS 2001 were taken from the U.S. Department of Education, NCES Common Core of Data (CCD), "Public Elementary/Secondary School Universe Survey," 1999–2000. 8 Since the measure of school poverty used for the United States in this analysis cannot be applied to other countries, only data for U.S. schools are used in these comparisons. 9 Informal initiatives to promote reading include book clubs, independent reading contests, and schoolwide recreational reading periods to encourage students to read. 10 Indicates that students are taught by a teacher with a teaching certificate. The NAEP reading assessment data from 1994 show that 95 percent of the teachers of fourth-grade students were certified in the state in which they taught. In the 2001 Schools and Staffing Survey (SASS), 97 percent of fourth-grade teachers reported that they were certified. 11 In the 2001 SASS, fourth-grade teachers reported that, on average, they had been teaching for 14 years.
Campbell, J.R., Kelly, D.L., Mullis, I.V.S., Martin, M.O., and Sainsbury, M. (2001). Framework and Specifications for PIRLS Assessment 2001. Chestnut Hill, MA: Boston College. Kapinus, B. (2003). PIRLS-IEA Reading Literacy Framework: Comparative Analysis of the 1991 IEA Reading Study and the Progress in International Reading Literacy Study (NCES 2003–05). U.S. Department of Education. Washington, DC: National Center for Education Statistics Working Paper. Martin, M.O., Mullis, I.V.S., and Kennedy, A.M. (2003). PIRLS 2001 Technical Report. Chestnut Hill, MA: Boston College.
|