Skip Navigation
small NCES header image


Research and Development Report

September 1998


 

 

Linking The National Assessment of
Educational Progress (NAEP)
and
The Third International Mathematics
and Science Study (TIMSS):
A Technical Report

 

Eugene G. Johnson, Educational Testing Service

Eugene Owen, Project Officer
National Center for Education Statistics

 


U.S. Department of Education
Office of Educational Research and Improvement
NCES 98-499

 

Adobe Acrobat PDF Download and View PDF Version of this Report (1,315K)

For high-quality printouts, please print from the PDF file.

 


U.S. Department of Education
Richard W. Riley
Secretary

Office of Educational Research and Improvement
C. Kent McGuire
Assistant Secretary

National Center for Education Statistics
Pascal D. Forgione, Jr.
Commissioner

Assessment Group
Gary W. Phillips
Associate Commissioner

 

National Center for Education Statistics

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries.

NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public.

We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or suggestions about this or any other NCES product or report, we would like to hear from you. Please direct your comments to:

National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
1900 K Street NW, Suite 9000
Washington DC 20006

The NCES World Wide Web Home Page is http://nces.ed.gov

Suggested Citation:

U.S. Department of Education, National Center for Education Statistics. Linking the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS): A Technical Report, NCES 98-499, by Eugene G. Johnson. Project Officer, Eugene Owen. Washington, DC: Government Printing Office, September 1998.

Contact:
Arnold Goldstein

 

Foreword


 

The Research and Development series of reports has been initiated for the following goals:

  1. To share studies and research that are developmental in nature. The results of such studies may be revised as the work continues and additional data become available.
  2. To share results of studies that are, to some extent, on the cutting edge of methodological developments. Emerging analytical approaches and new computer software development often permit new, and sometimes controversial, analysis to be done. By participating in "frontier research," we hope to contribute to the resolution of issues and improved analysis.
  3. To participate in discussions of emerging issues of interest to educational researchers, statisticians, and the Federal statistical community in general. Such reports may document workshops and symposiums sponsored by the National Center for Education Statistics (NCES) that address methodological and analytical issues or may share and discuss issues regarding NCES practice, procedures, and standards.

The common theme in all three goals is that these reports present results or discussions that do not reach definitive conclusions at this point in time, either because the data are tentative, the methodology is new and developing, or the topic is one on which there are divergent views. Therefore, the techniques and inferences made from the data are tentative and are subject to revision. To facilitate the process of closure on the issues, we invite comment, criticism, and alternatives to what we have done. Such responses should be directed to:

 

Marilyn M. McMillen
Chief Statistician
Statistical Standards Services Group
National Center for Education Statistics
1900 K Street NW, Suite 9000
Washington DC 20006

 

 

 

Acknowledgments



This report benefited greatly from the contributions of many people.

My plans about linking NAEP and TIMSS and estimating the components of variance of the link were greatly clarified through conversations with Robert Mislevy, Don McLaughlin, and Juliet Shaffer, all of whom also made insightful comments on successive drafts of the report. Helpful comments on the report were also made by Susan Ahmed, Albert Beaton, James Chromy, Jon Cohen, John Dossey, Paul Holland, Richard Jaeger, Lyle Jones, Wayne Martin, Ingram Olkin, Senta Raizen, Douglas Rindone, Keith Rust, David Thissen, and Valerie Williams.

Data analysis was led by Minhwei Wang, with significant contributions by David Freund, Xuefei Hui, Edward Kulick, and Steve Wang. Gerry Kokolis and Jennifer Nelson created the graphics in the report. Sincere thanks also go to Eugenio Gonzalez and Stephen Roey for providing the needed TIMSS data.

This report was funded through the National Center for Education Statistics (NCES) in the Office of Educational Research and Improvement of the U.S. Department of Education. Pascal Forgione, NCES Commissioner, provided support and guidance. Advice and guidance was collegially provided by Mary Frase, Arnold Goldstein, Marilyn McMillen, Martin Orland, Eugene Owen, Lois Peak, and Gary Phillips. I am also grateful to Ramsey Selden, Karol Krotki, and Patrick Gonzales of the Education Statistics Services Institute for their help.

Thanks to Adriene Siegendorf of the Education Statistics Services Institute, and my co-author of the Results Report, for the tables comparing the states to the nations.

Special thanks go to Nancy Caldwell of Westat for her support, encouragement, and insight throughout the process of creating this report.

Finally, thanks go to Susan Crawford of Westat and Janet Johnson formerly of the Educational Testing Service for editorial support, and a particular thanks to Joan Murphy of Westat, who edited and oversaw the production of the many drafts, up to and including the final report.

Eugene Johnson

 

 

Table of Contents



Chapter

PREFACE

FOREWORD

ACKNOWLEDGMENTS

1 INTRODUCTION

2 NAEP AND TIMSS DATA

3 TYPES OF LINKAGE

4 ESTABLISHING THE LINK

5 VARIANCE OF THE LINKING FUNCTION

6 TOTAL VARIANCE OF THE LINKING FUNCTION

7 LINKING FUNCTION FOR THE PUBLIC SCHOOL STATE DATA

8 LINKING FUNCTION FOR INTERNATIONAL MARKER LEVELS

9 VALIDATION

10 RESULTS

11 CONCLUSIONS

REFERENCES

 


List of Appendixes

Appendix

A Validation Studies of the Linkage between NAEP and TIMSS Eighth Grade Mathematics Assessments Validation Studies of the Linkage between NAEP and TIMSS Eighth Grade Science Assessments

B Summary of Deviations from Study Guidelines

C Comparisons of each NAEP State and Jurisdiction with the TIMSS Nations For Grade 8 Mathematics

D Comparisons of each NAEP State and Jurisdiction with the TIMSS Nations for Grade 8 Science

 

List of Tables

Table

1 Parameter estimates for the linking of NAEP to TIMSS for grade 8 font>

2 Values of and x used for comparing variances of the linked estimate for grade 8

3 Components of due to sampling for grade 8

4 Comparison of the naive estimate of with the estimate including sampling error for grade 8

5 Components of due to measurement error for grade 8

6 Comparison of the estimate of before and after including measurement error for grade 8

7A Parameters and linked estimates derived within subpopulation—grade 8 mathematics

7B Parameters and linked estimates derived within subpopulation—grade 8 science

8 Comparison of the component of variance due to model misspecification estimated by with its expected value estimated by for grade 8

9 Value of the component of due to temporal shift for grade 8

10 Total variance of with percentages due to components for grade 8

11 Parameter estimates for the linking of public school NAEP to TIMSS for grade 8

12 Components of for the public school link for grade 8

13 International marker levels of achievement for grade 8

14 Predicted NAEP cutpoints and their standard errors corresponding to the TIMSS marker levels—public school linking for grade 8

15 Comparison of actual TIMSS mean proficiency with predicted TIMSS proficiency from NAEP results (data are from public schools only) for grade 8

16 Ninety-five percent confidence intervals for the percentages above the TIMSS marker levels based on actual TIMSS data and on predictions from NAEP (data are from public schools only) for grade 8

17 Estimated TIMSS scores from public school, 1996 NAEP data for states and jurisdictions: Grade 8 mathematics and science

18 Actual 1995 TIMSS scores for countries: Grade 8 mathematics and science

19 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in mathematics at grade 8 based on estimates from 1996 NAEP public school data for states and jurisdictions

20 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in science at grade 8 based on estimates from 1996 NAEP public school data for states and jurisdictions

21 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in mathematics at grade 8 based on actual 1995 TIMSS data for countries

22 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in science at grade 8 based on actual 1995 TIMSS data for countries

 


List of Figures

Figure

1 Marginal distributions for pairs of hypothetical tests

2 Scatter plots of the hypothetical test score pairs

3 Rootograms comparing proficiency distributions for 1995 TIMSS and 1996 NAEP (NAEP distributions adjusted to have same mean and standard deviation as TIMSS)


Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education