Skip Navigation
Illustration/Logo View Quarterly by  This Issue  |  Volume and Issue  |  Topics
Education Statistics Quarterly
Vol 5, Issue 2, Topic: International Statistics
International Comparisons in Fourth-Grade Reading Literacy: Findings From the Progress in International Reading Literacy Study (PIRLS) of 2001
By: Laurence T. Ogle, Anindita Sen, Erin Pahlke, Leslie Jocelyn, David Kastberg, Stephen Roey, and Trevor Williams
 
This article was excerpted from the Statistical Analysis Report of the same name. The sample survey data are primarily from the Progress in International Reading Literacy Study (PIRLS).
 
 

The Progress in International Reading Literacy Study of 2001 (PIRLS 2001) is an assessment of reading comprehension conducted by the International Association for the Evaluation of Educational Achievement (IEA). Thirty-five countries assessed the reading literacy of students in the upper of the two grades with the most 9-year-olds (fourth grade in most countries, including the United States). PIRLS 2001 provides comparative information on the reading literacy of these fourth-graders and also examines factors that may be associated with the acquisition of reading literacy in young children.

PIRLS 2001 will help educators and policymakers by answering questions such as the following:

  • How well do fourth-grade students read?
  • How do students in one country compare with students in another country?
  • Do fourth-grade students value and enjoy reading?
  • Internationally, how do the reading habits and attitudes of students vary?

As the sponsor for PIRLS 2001 in the United States, the National Center for Education Statistics (NCES) is reporting findings from the study that compare the United States with other countries and that take a closer look at performance within the United States. The full report on the international study is available at www.pirls.org. Also available at this site is the PIRLS 2001 Technical Report (Martin, Mullis, and Kennedy 2003), which examines specific technical issues related to the assessment. Supporting data for the tables and analyses in this report are available at nces.ed.gov/surveys/pirls.

Background

PIRLS 2001 follows by 10 years a prior IEA study of reading literacy called the IEA International Reading Literacy Study of 1991. Over the 10 years between these studies, progress has been made in the ways in which students are assessed and in the construction of the assessment instruments themselves. There has also been a shift in the design of the assessments. Thus, while PIRLS 2001 can trace its evolution from the 1991 IEA study, it is nevertheless a different study.

PIRLS 2001 is the first in a planned 5-year cycle of international trend studies in reading literacy by the IEA. PIRLS is designed to assist participating countries in monitoring the reading literacy of their fourth-grade populations in comparison to other countries.

Construction and administration

A group of distinguished international reading scholars, the Reading Development Group, was formed to construct the PIRLS 2001 Framework (see Campbell et al. 2001) and endorse the final reading assessment. Each country followed internationally prescribed procedures to ensure valid translations and representative samples of students. Quality Control Monitors were then appointed in each country to monitor the testing sessions at the schools to ensure that the high standards of the PIRLS 2001 data collection process were met.

Reading literacy achievement was measured by using a selection of four literary passages drawn from children's storybooks and four informational texts. Submitted and reviewed by the PIRLS 2001 countries, the literary passages included realistic stories and traditional tales. The informational texts included chronological and nonchronological articles, a biographical article, and an informational leaflet.

Data collection

Data were collected in the final months of the 2000–01 school year. In the United States, data were collected in the spring of 2001 from both public and private schools.

Definition and aspects of reading literacy

PIRLS 2001 measures reading abilities at a time in students' schooling when most have learned how to read and are now using reading to learn.

PIRLS 2001 defines reading literacy as follows:

The ability to understand and use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers, and for enjoyment (Campbell et al. 2001, p. 3).
In PIRLS 2001, three aspects of reading literacy are assessed: purposes of reading, processes of comprehension, and reading behavior and attitudes. The first two aspects of reading literacy form the basis of the written test of reading comprehension, while the student background questionnaire addresses the third aspect.

Purposes of reading refers to the two types of reading that account for most of the reading young students do, both in and out of school: (1) reading for literary experience, and (2) reading to acquire and use information. In the assessment, narrative fiction is used to assess students' ability to read for literary experience, while a variety of informational texts are used to assess students' ability to acquire and use information while reading. The PIRLS 2001 assessment contains an equal proportion of text assessing each purpose.

Processes of comprehension refers to ways in which readers construct meaning from the text. Readers (1) focus on and retrieve specific ideas, (2) make inferences, (3) interpret and integrate ideas and information, and (4) examine or evaluate text features. As shown in figure A, each process is assessed within each purpose of reading.

Figure A. Percentage of PIRLS assessment items devoted to reading purposes and processes
Figure A. Percentage of PIRLS assessment items devoted to reading purposes and processes

NOTE: Detail may not sum to totals due to rounding.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 2 on p. 3 of the complete report from which this article is excerpted.)

Back to Top


PIRLS 2001 scores are reported on a scale of 0 to 1000, with an international average of 500 and a standard deviation of 100.1 For the 35 countries that participated in PIRLS 2001, figure B presents the average scores for three scales: the combined reading literacy scale and its two components, the literary and informational subscales.2 The average scores of U.S. students are compared to the average scores of students in other participating countries and the international average score.3

U.S. student performance on the combined reading literacy scale
  • U.S. fourth-grade students perform significantly better than the international average of 500 on the combined reading literacy scale.
  • U.S. fourth-graders outperform their counterparts in 23 of the 34 other countries participating in PIRLS 2001, although they score lower than students in England, the Netherlands, and Sweden. No detectable differences in scores are found between U.S. students and their counterparts in eight of the remaining PIRLS 2001 countries.
U.S. student performance on subscales
  • U.S. fourth-grade students perform better than the international averages on both of the reading subscales.
  • Sweden outscores the United States on the literary subscale, and five countries—Bulgaria, England, Latvia, the Netherlands, and Sweden—outperform the United States on the informational subscale.
  • U.S. fourth-graders outscore students in 26 countries on the literary subscale and outperform their counterparts in 17 countries on the informational subscale.

Figure B. Fourth-graders' average scores for the combined reading literacy scale, literary subscale, and informational subscale, by country: 2001
Figure B. Fourth-graders' average scores for the combined reading literacy scale, literary subscale, and informational subscale, by country: 2001

1Met guidelines for sample participation rates only after replacement schools were included.

2National Defined Population covers less than 95 percent of National Desired Population.

3National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent.

4Canada is represented by the provinces of Ontario and Quebec (O, Q) only.

5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China.

6National Defined Population covers less than 80 percent of National Desired Population.

7Nearly satisfied guidelines for sample participation rates after replacement schools were included.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 3 on p. 5 of the complete report from which this article is excerpted.)


The average scores for reading literacy describe how a country performs overall compared to other nations, but they provide no information about the way scores are distributed within the countries. One country with an average score similar to another could have large numbers of high- and low-scoring students, while the other country could have large numbers of students performing at about the average score. Figure C details how scores are distributed across countries.

  • In the United States, the 5th percentile score for combined reading literacy is 389. Ninety-five percent of U.S. students score above 389; in the same way, 5 percent of students score above 663, the 95th percentile score. This means that the top 5 percent of U.S. students score at least 274 points higher than the bottom 5 percent.
Looking at the length of the bars in figure C gives a sense of how large the differences are between a country's highest and lowest performing students, but it does not describe how many students are high or low performing. As with average scores, because of the statistical techniques used to sample students, it is not accurate to rank countries' scoring variation based simply on the length of the bars shown in figure C. Standard deviations of the combined reading literacy average scores give a mathematical way to tell how greatly scores are spread out from the country's average score.
  • Seventeen countries, or about half of the countries participating in PIRLS 2001, show less variation in student performance than the United States. Ten countries show more variation, while the remaining eight countries show no detectable differences in variation in student performance compared to the United States.

Back to Top

Average scores in figure B indicate how well the United States performs relative to other countries, but the scores do not indicate the proficiency required to reach a particular score. To gain a better understanding of what scores represent in terms of reading proficiency, PIRLS 2001 selected four cutoff points on the combined reading literacy scale labeled international benchmarks. These benchmarks were selected to correspond to the score points at or above which the lower quarter, median, upper quarter, and top 10 percent of fourth-graders in the international PIRLS 2001 sample performed.4

Student responses at the four benchmarks were analyzed to describe a set of reading skills and strategies displayed by fourth-graders at those points. These descriptions, together with the cut point scores, are listed in figure D.5

  • On the combined reading literacy scale, 19 percent of the fourth-grade students in the United States reach the top 10 percent benchmark, 41 percent the upper quarter benchmark, 68 percent the median benchmark, and 89 percent the lower quarter benchmark. The percentage of U.S. fourth-graders reaching each of these benchmarks is higher than the international averages.
  • Compared to the United States, no other country but England (24 percent) reports a higher percentage of students at the top 10 percent benchmark on the combined reading literacy scale. Sweden (47 percent) reports a higher share of students at the upper quarter benchmark compared to the United States.
  • On the literary subscale, for the United States, 22 percent of students reach the top 10 percent benchmark, 43 percent the upper quarter benchmark, 70 percent the median benchmark, and 90 percent the lower quarter benchmark. The percentage of U.S. fourth-graders reaching each of these benchmarks on the literary subscale is higher than the corresponding international averages.
  • On the informational subscale, for the United States, 15 percent of students reach the top 10 percent benchmark, 36 percent the upper quarter benchmark, 66 percent the median benchmark, and 89 percent the lower quarter benchmark. The percentage of U.S. fourth-graders reaching these benchmarks on the informational subscale is higher than the corresponding international averages.
Figure C. Distribution of average combined reading literacy scale scores of fourth-graders by percentiles, by country: 2001
Figure C. Distribution of average combined reading literacy scale scores of fourth-graders by percentiles, by country: 2001
1Met guidelines for sample participation rates only after replacement schools were included.

2National Defined Population covers less than 95 percent of National Desired Population.

3National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent.

4Canada is represented by the provinces of Ontario and Quebec (O, Q) only.

5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China.

6National Defined Population covers less than 80 percent of National Desired Population.

7Nearly satisfied guidelines for sample participation rates after replacement schools were included.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 4 on p. 7 of the complete report from which this article is excerpted.)

Figure D. Fourth-graders' reading skills and strategies, and cut point scores, by benchmark points for the combined reading literacy scale: 2001
Figure D. Fourth-graders' reading skills and strategies, and cut point scores, by benchmark points for the combined reading literacy scale: 2001

1The responses of students who score within 5 points of each of the cut point scores were evaluated to determine reading skills and strategies displayed by fourth-graders at those points. Procedures used for anchoring these items to the benchmarks are explained more fully in the PIRLS Technical Report at www.pirls.org.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 5 on p. 8 of the complete report from which this article is excerpted.)

Back to Top


Achievement by sex

In the United States and many other countries, policymakers and educators are interested not only in overall achievement but also in achievement by specific groups of students. For example, patterns of differences between boys and girls in reading achievement across countries can point to areas where additional educational resources might be focused.

  • Fourth-grade girls score higher than fourth-grade boys on the combined reading literacy scale, on average, in every participating PIRLS 2001 country (figure E). In the United States, on average, girls score 18 points higher than boys on the combined reading literacy scale. Internationally, the average score difference between boys and girls ranges from 8 points (Italy) to 27 points (Belize, Iran, and New Zealand).6
  • Fourth-grade girls score higher than boys on both the literary and informational subscales in all of the participating PIRLS 2001 countries. In the United States, fourth-grade girls, on average, outscore boys by 16 points on both the literary and informational subscales.
  • Fourth-grade girls in Sweden, England, the Netherlands, and Bulgaria outscore U.S. girls on the combined reading literacy scale. However, U.S. girls perform better than their counterparts in 21 of the participating PIRLS 2001 countries.
  • Fourth-grade boys in the Netherlands and Sweden outperform U.S. boys on the combined reading literacy scale, although U.S. boys perform better than their peers in 22 of the participating PIRLS 2001 countries.

U.S. achievement by race/ethnicity

Another area of interest among policymakers and educators is the achievement of racial/ethnic groups. A number of countries that participated in PIRLS 2001 have large and diverse racial/ethnic groups. However, since these groups vary considerably across countries, it is not possible to compare their performance internationally. Thus, the findings in this section refer only to PIRLS 2001 results for the United States.

  • With the exception of Black fourth-graders, each racial/ethnic group in the United States scores higher than the international average (i.e., 500) on the combined reading literacy scale, as well as on the two reading subscales.
  • There is considerable variation in scores among the racial/ethnic groups in the United States. On average, White fourth-grade students perform better than Black and Hispanic fourth-graders on the combined reading literacy scale, as well as on the two subscales (figure F). Asian fourth-grade students, on average, also perform better than Black and Hispanic students on the combined reading literacy scale, as well as on the informational subscale. On the literary subscale, Asian students perform better than Black students, while there are no detectable differences in performance between Asian and Hispanic students. There are no detectable differences in scores between White and Asian fourth-grade students across any of the reading scales.
  • A larger percentage of White fourth-graders in the United States reach the top 10 percent benchmark on the combined reading literacy scale than do Black or Hispanic fourth-graders. Thus, 25 percent of White fourth-graders reach the top 10 percent benchmark, while 6 percent of Black and 10 percent of Hispanic fourth-graders reach the same benchmark. There is no detectable difference in the percentages of White and Asian fourth-graders who reach the top 10 percent benchmark, but a larger percentage of Asian fourth-graders reach this benchmark than do Black fourth-graders.
  • A larger percentage of both White and Asian fourth-graders in the United States reach the upper quarter benchmark on the combined scale than do Black and Hispanic fourth-graders. Thus, 51 percent of White and 46 percent of Asian fourth-graders reach the upper quarter benchmark, while 19 percent of Black and 27 percent of Hispanic fourth-graders reach the same benchmark.

U.S. achievement by control of school

On average, fourth-grade students in private schools in the United States score significantly higher than fourth-grade students in public schools on the combined reading literacy scale, and also on the literary and informational subscales. For example, on the combined reading literacy scale and the informational subscale, on average, fourth-grade students in private schools score 42 points higher than students in public schools. On the literary subscale, private school fourth-graders score an average of 45 points higher than public school fourth-graders.

U.S. achievement by poverty level in public schools

One measure of poverty in U.S. public elementary schools is the percentage of students eligible for free or reduced-price lunch.7 In order to examine how fourth-graders' scores on the combined reading literacy scale are associated with their schools' poverty level (percentage of students receiving free or reduced-price lunch), U.S. public schools were classified into five groups: (1) schools with the lowest poverty levels of less than 10 percent; (2) schools with poverty levels ranging from 10 to 24.9 percent; (3) schools with poverty levels ranging from 25 to 49.9 percent; (4) schools with poverty levels ranging from 50 to 74.9 percent; and (5) schools with the highest poverty levels of 75 percent or more.8

  • Fourth-graders in U.S. public elementary schools with the highest poverty levels score lower on the combined reading literacy scale compared to their counterparts in schools with lower poverty levels.
  • Fourth-graders in schools with intermediate poverty levels of 10 to 24.9 percent and 25 to 49.9 percent score higher on the combined reading literacy scale than students in schools with poverty levels of 50 to 74.9 percent and 75 percent or more. However, there are no detectable differences in scores between U.S. fourth-graders in public schools with poverty levels of 10 to 24.9 percent and 25 to 49.9 percent.
  • On average, lower percentages of fourth-graders in the highest poverty public schools in the United States reach the upper two international benchmarks (top 10 percent and upper quarter) than their counterparts in the lowest poverty schools. For example, in the highest poverty schools, about 3 percent of fourth-grade students reach the top 10 percent international benchmark, while in the lowest poverty schools, about 34 percent of fourth-grade students reach the same benchmark. Additionally, about 14 percent of students in the highest poverty schools reach the upper quarter benchmark, but in the lowest poverty schools, 64 percent of students reach that benchmark.
Figure E. Difference in average scores between boys and girls for the combined reading literacy scale of fourth-graders, by country: 2001
Figure E. Difference in average scores between boys and girls for the combined reading literacy scale of fourth-graders, by country: 2001
1National Desired Population does not cover all of International Desired Population because coverage falls below 65 percent.

2Canada is represented by the provinces of Ontario and Quebec (O, Q) only.

3Met guidelines for sample participation rates only after replacement schools were included.

4National Defined Population covers less than 95 percent of National Desired Population.

5Hong Kong is a Special Administrative Region (SAR) of the People's Republic of China.

6National Defined Population covers less than 80 percent of National Desired Population.

7Nearly satisfied national guidelines for sample participation rates after replacement schools were included.

NOTE: All average score differences reported are statistically significant.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 7 on p.11 of the complete report from which this article is excerpted.)

Figure F. U.S. fourth-graders' average scores for the combined reading literary scale, literacy subscale, and informational subscale, by race/ethnicity: 2001
Figure F. U.S. fourth-graders' average scores for the combined reading literary scale, literacy subscale, and informational subscale, by race/ethnicity: 2001

NOTE: Black includes African American, and Hispanic includes Latino. Race categories exclude Hispanic origin unless specified. The United States met guidelines for sample participation rates after replacement schools were included.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 9 on p. 13 of the complete report from which this article is excerpted.)

Back to Top


Reading curriculum and instructional time

Do school principals and teachers encourage reading instruction through a variety of initiatives? What proportion of the school day is spent in reading instruction? Answers to these questions can give an indication of the emphasis that reading instruction receives in the curriculum of a country.

  • According to school principals, 72 percent of U.S. fourth-graders attend schools that have a written statement describing the reading curriculum, which is nearly double the international average of 37 percent.
  • Almost all U.S. fourth-grade students (95 percent) attend schools with a curricular emphasis on reading. This is greater than the international average of 78 percent.
  • Principals report that 95 percent of U.S. fourth-grade students attend schools with informal initiatives to encourage reading, which is greater than the international average of 76 percent.9
  • Based on teacher reporting, 65 percent of U.S. fourth-graders receive more than 6 hours of reading instruction per week, a higher percentage than the international average of 28 percent (figure G). This percentage is also higher than the national average in 31 of the other 34 participating PIRLS 2001 countries.
  • The average combined reading literacy achievement scores of U.S. fourth-graders do not vary by the amount of instructional time they receive.

Teacher preparation and experience

Examining teachers' preparation and tenure indicates the experience of teachers in the classroom. On the teacher questionnaire in PIRLS 2001, teachers were asked about the training they have received and the number of years they have been teaching.

  • Based on teacher reports of their preparation for teaching, 95 percent of U.S. fourth-graders are taught by certified teachers.10 This is higher than the corresponding international average of 89 percent.
  • U.S. fourth-graders appear to be taught by teachers who have more experience teaching fourth grade than their counterparts in the majority of the participating PIRLS 2001 countries. On average, U.S. fourth-grade students are taught by teachers who have been teaching fourth grade for 7 years.11 Twenty-six of the other 34 participating countries reported that their fourth-graders are taught by teachers with fewer years of experience teaching fourth grade.

Figure G. Percentage of fourth-graders by average number of hours of reading instruction each week: 2001
Figure G. Percentage of fourth-graders by average number of hours of reading instruction each week: 2001

1Significant difference between U.S. average and international average in this category.

NOTE: The United States met guidelines for sample participation rates after replacement schools were included.

SOURCE: International Association for the Evaluation of Educational Achievement, Progress in International Reading Literacy Study (PIRLS), 2001. (Originally published as figure 11 on p.16 of the complete report from which this article is excerpted.)

Back to Top

Reading outside of school for enjoyment

To investigate the reading habits of fourth-graders outside of school, PIRLS asked students a series of questions about whether they read for fun outside of school and how often they did so. Students could indicate that they read for fun "every day or almost every day," "once or twice a week," "once or twice a month," or "never or almost never."

  • Thirty-five percent of U.S. fourth-graders report reading for fun every day or almost every day. This percentage is smaller than the international average of 40 percent.
  • Thirty-two percent of U.S. fourth-graders report that they never or almost never read for fun outside of school, a significantly higher percentage than the international average of 18 percent.
  • In the United States, fourth-graders who read for fun every day or almost every day have higher average scores on the combined reading literacy scale compared to those who never or almost never read for fun, or do so once or twice a month. This pattern holds at the international level as well, based on the international averages.
Choice of activities outside of school

To learn more about students' reading habits, PIRLS 2001 asked students about their choice of reading materials and how often they read different types of texts when they are not in school.

  • In the United States, 92 percent of fourth-graders report reading for information at least once or twice a month, a higher percentage than those who report reading either literary fiction, such as stories or novels (79 percent), or comics (43 percent) at least once or twice a month.
  • In the United States, 43 percent of fourth-graders report that they read comics at least once or twice a month, a significantly lower percentage than the international average of 74 percent.
  • U.S. fourth-graders who report reading literary fiction outside of school at least once or twice a month have higher scores on the combined reading literacy scale than those who never or almost never do so. This pattern is also evident at the international level, based on international averages.
  • No measurable differences in scores on the combined reading literacy scale are detected between U.S. fourth-graders who read informational materials every day or almost every day, and those who never or almost never do so.

PIRLS 2001 also asked students about their TV- and video-watching habits.

  • Eighteen percent of U.S. fourth-graders report watching TV or videos on a normal school day for 5 hours or more. This is significantly higher than the international average of 12 percent. On average, U.S. fourth-graders report watching TV or videos daily for a greater number of hours than the international average (2.2 hours vs. 2 hours).
  • Looking at the international average for the combined reading literacy scale, fourth-graders who watch TV for more than 5 hours on a normal school day score lower than those who watch TV for 3 to 5 hours a day or less frequently. In the United States, the same finding holds.

The sample items presented here show actual student responses and compare U.S. fourth-graders' performance to the international average. The items also demonstrate acceptable performance at the four benchmarks (top 10 percent, upper quarter, median, and lower quarter). The reading passage (exhibit A) and all of these items have been released to the public by IEA.

Exhibit A. One of the reading passages used in PIRLS 2001

Exhibit A. One of the reading passages used in PIRLS 2001
Exhibit A. One of the reading passages used in PIRLS 2001

SOURCE: Previously published on p. 20 of the complete report from which this article is excerpted.

Sample item at the top 10 percent PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Sample item at the top 10 percent PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Sample item at the upper quarter PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Sample item at the median PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Sample item at the lower quarter PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Sample item at the lower quarter PIRLS 2001 international benchmark, with response illustrating performance at this benchmark

Back to Top


Reading performance over time

Ten years before PIRLS 2001 was administered, the IEA conducted the IEA International Reading Literacy Study of 1991. This study, like PIRLS 2001, assessed the reading literacy of fourth-graders in over 30 countries using 42 items taken from 6 reading passages. However, when a follow-up for the 1991 study was being planned, the IEA decided to discontinue it and develop a new assessment incorporating the latest approaches to measuring reading literacy (Campbell et al. 2001). This new study would become PIRLS 2001.

In anticipation of the simultaneous release of PIRLS 2001 and the IEA International Reading Literacy Study of 1991, NCES commissioned a comparative analysis of the two assessments. Frameworks, passages, and items in both studies were reviewed and compared. Results indicate that the two studies are quite different. To cite a few examples: Reading passages in PIRLS 2001 were found to be "longer, more engaging, and more complex in most cases" than those found in the IEA International Reading Literacy Study of 1991 (Kapinus 2003, p. 8). PIRLS 2001 also used many more constructed-response (essay-type) questions and presented them in a way "that might have improved students' motivation to read and respond to the texts" (Kapinus 2003, p. 8). The analysis also found that, in general, PIRLS 2001 tapped skills "requiring deeper thinking" than those in the IEA International Reading Literacy Study of 1991 (Kapinus 2003, p. 8). Because of these and other differences, it is impossible to directly compare results from these two assessments. However, separately, each study provides important clues about how well students in these countries, including U.S. fourth-graders, perform in reading literacy.

While participating in PIRLS 2001, some countries expressed interest in comparing reading performance between 1991 and 2001. Since comparisons between the two assessments were impossible, the IEA gave participating countries an opportunity to readminister the 1991 study during the PIRLS 2001 administration. This readministered study was identical in content, timing, and directions to that given to students in 1991 and allowed comparisons of the performance of students in 2001 with those in 1991. A separate sample of students was drawn in each country so as not to overburden students assessed in PIRLS 2001. Nine countries, including the United States, participated in the 2001 readministration of the IEA International Reading Literacy Study of 1991.

Performance on the IEA International Reading Literacy Study of 1991
  • Based on the readministration of the 1991 study in 2001, no detectable change is observed in the achievement of fourth-graders on the combined reading literacy scale in the United States in 2001 compared to 1991.
  • Fourth-graders in five of the nine participating countries perform significantly better, on average, on the 1991 study combined reading literacy scale in 2001 compared to 1991, while fourth-graders in three countries show no detectable difference in average achievement between 1991 and 2001. One country, Sweden, has a significantly lower average score in 2001 than in 1991.
Back to Top


Footnotes

1 The international average is the mean of all countries participating in the study calculated so that all participating countries have the same contribution to the average. The PIRLS 2001 scale average for each scale (the combined reading literacy scale and the literary and informational subscales) across countries was set to 500 and the standard deviation to 100.

2 Average scores for each country are based on a sample of students, rather than all students, and are estimates of the population value of all 9-year-olds in each country. The combined literacy scale is based on the distribution of scores on all the test items, while the subscales are based on only the items that belong to each subscale. Hence, the combined reading literacy score is not the statistical average of the scores of the two subscales.

3 No statistical adjustments (such as Bonferonni) are made while carrying out multiple comparisons between the United States and other countries. In order to be consistent with the comparisons carried out for the

4 Benchmarking in PIRLS describes the performance of students at four international benchmarks based on the distribution of scores and the pattern of items answered correctly. Proficiency levels for the National Assessment of Educational Progress (NAEP) (i.e., Basic, Proficient, and Advanced ) are established by the National Assessment Governing Board based on recommendations from broadly representative panels of educators and the general public who determine what students should know and be able to do at the three levels of performance in each subject area and in each grade assessed.

5 If students' reading achievement was distributed in the same way in every country, then each country would be expected to have approximately 10 percent of fourth-graders reaching the top 10 percent benchmark, 25 percent the upper quarter benchmark, 50 percent the median benchmark, and 75 percent the lower quarter benchmark.

6 Differences in scores by sex are not shown here for Kuwait due to low response rates on the question related to sex. However, the international average includes Kuwait's average scale score.

7 Data for the percentage of students eligible for free or reduced-price lunch in U.S. public elementary schools participating in PIRLS 2001 were taken from the U.S. Department of Education, NCES Common Core of Data (CCD), "Public Elementary/Secondary School Universe Survey," 1999–2000.

8 Since the measure of school poverty used for the United States in this analysis cannot be applied to other countries, only data for U.S. schools are used in these comparisons.

9 Informal initiatives to promote reading include book clubs, independent reading contests, and schoolwide recreational reading periods to encourage students to read.

10 Indicates that students are taught by a teacher with a teaching certificate. The NAEP reading assessment data from 1994 show that 95 percent of the teachers of fourth-grade students were certified in the state in which they taught. In the 2001 Schools and Staffing Survey (SASS), 97 percent of fourth-grade teachers reported that they were certified.

11 In the 2001 SASS, fourth-grade teachers reported that, on average, they had been teaching for 14 years.

Back to Top


Campbell, J.R., Kelly, D.L., Mullis, I.V.S., Martin, M.O., and Sainsbury, M. (2001). Framework and Specifications for PIRLS Assessment 2001. Chestnut Hill, MA: Boston College.

Kapinus, B. (2003). PIRLS-IEA Reading Literacy Framework: Comparative Analysis of the 1991 IEA Reading Study and the Progress in International Reading Literacy Study (NCES 2003–05). U.S. Department of Education. Washington, DC: National Center for Education Statistics Working Paper.

Martin, M.O., Mullis, I.V.S., and Kennedy, A.M. (2003). PIRLS 2001 Technical Report. Chestnut Hill, MA: Boston College.

Back to Top


Data sources: The Progress in International Reading Literacy Study (PIRLS), 2001; the International Association for the Evaluation of Educational Achievement (IEA) International Reading Literacy Study of 1991; and the NCES Common Core of Data (CCD), "Public Elementary/Secondary School Universe Survey," 1999–2000.

For technical information, see the complete report:

Ogle, L.T., Sen, A., Pahlke, E., Jocelyn, L., Kastberg, D., Roey, S., and Williams, T. (2003). International Comparisons in Fourth-Grade Reading Literacy: Findings From the Progress in International Reading Literacy Study (PIRLS) of 2001 (NCES 2003–073).

Author affiliations: L.T. Ogle, NCES; A. Sen and E. Pahlke, Education Statistics Services Institute; and L. Jocelyn, D. Kastberg, S. Roey, and T. Williams, Westat.

For questions about content, contact Laurence Ogle (laurence.ogle@ed.gov).

To obtain the complete report (NCES 2003–073), call the toll-free ED Pubs number (877–433–7827) or visit the NCES Electronic Catalog (http://nces.ed.gov/pubsearch).


back to top