Research and Development Report
Linking The National Assessment of
Eugene G. Johnson, Educational Testing Service
Eugene Owen, Project Officer
Download and View PDF Version of this Report (1,315K)
For high-quality printouts, please print from the PDF file.
U.S. Department of Education
The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries.
NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public.
We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or suggestions about this or any other NCES product or report, we would like to hear from you. Please direct your comments to:
The NCES World Wide Web Home Page is http://nces.ed.gov
U.S. Department of Education, National Center for Education Statistics. Linking the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS): A Technical Report, NCES 98-499, by Eugene G. Johnson. Project Officer, Eugene Owen. Washington, DC: Government Printing Office, September 1998.
The Research and Development series of reports has been initiated for the following goals:
The common theme in all three goals is that these reports present results or discussions that do not reach definitive conclusions at this point in time, either because the data are tentative, the methodology is new and developing, or the topic is one on which there are divergent views. Therefore, the techniques and inferences made from the data are tentative and are subject to revision. To facilitate the process of closure on the issues, we invite comment, criticism, and alternatives to what we have done. Such responses should be directed to:
Marilyn M. McMillen
This report benefited greatly from the contributions of many people.
My plans about linking NAEP and TIMSS and estimating the components of variance of the link were greatly clarified through conversations with Robert Mislevy, Don McLaughlin, and Juliet Shaffer, all of whom also made insightful comments on successive drafts of the report. Helpful comments on the report were also made by Susan Ahmed, Albert Beaton, James Chromy, Jon Cohen, John Dossey, Paul Holland, Richard Jaeger, Lyle Jones, Wayne Martin, Ingram Olkin, Senta Raizen, Douglas Rindone, Keith Rust, David Thissen, and Valerie Williams.
Data analysis was led by Minhwei Wang, with significant contributions by David Freund, Xuefei Hui, Edward Kulick, and Steve Wang. Gerry Kokolis and Jennifer Nelson created the graphics in the report. Sincere thanks also go to Eugenio Gonzalez and Stephen Roey for providing the needed TIMSS data.
This report was funded through the National Center for Education Statistics (NCES) in the Office of Educational Research and Improvement of the U.S. Department of Education. Pascal Forgione, NCES Commissioner, provided support and guidance. Advice and guidance was collegially provided by Mary Frase, Arnold Goldstein, Marilyn McMillen, Martin Orland, Eugene Owen, Lois Peak, and Gary Phillips. I am also grateful to Ramsey Selden, Karol Krotki, and Patrick Gonzales of the Education Statistics Services Institute for their help.
Thanks to Adriene Siegendorf of the Education Statistics Services Institute, and my co-author of the Results Report, for the tables comparing the states to the nations.
Special thanks go to Nancy Caldwell of Westat for her support, encouragement, and insight throughout the process of creating this report.
Finally, thanks go to Susan Crawford of Westat and Janet Johnson formerly of the Educational Testing Service for editorial support, and a particular thanks to Joan Murphy of Westat, who edited and oversaw the production of the many drafts, up to and including the final report.
5 VARIANCE OF THE LINKING FUNCTION
5.1 Component of Due to Sampling
6 TOTAL VARIANCE OF THE LINKING FUNCTION
7 LINKING FUNCTION FOR THE PUBLIC SCHOOL STATE DATA
8 LINKING FUNCTION FOR INTERNATIONAL MARKER LEVELS
List of Appendixes
A Validation Studies of the Linkage between NAEP and TIMSS Eighth Grade Mathematics Assessments Validation Studies of the Linkage between NAEP and TIMSS Eighth Grade Science Assessments
B Summary of Deviations from Study Guidelines
C Comparisons of each NAEP State and Jurisdiction with the TIMSS Nations For Grade 8 Mathematics
D Comparisons of each NAEP State and Jurisdiction with the TIMSS Nations for Grade 8 Science
List of Tables
1 Parameter estimates for the linking of NAEP to TIMSS for grade 8 font>
2 Values of and x used for comparing variances of the linked estimate for grade 8
3 Components of due to sampling for grade 8
4 Comparison of the naive estimate of with the estimate including sampling error for grade 8
5 Components of due to measurement error for grade 8
6 Comparison of the estimate of before and after including measurement error for grade 8
7A Parameters and linked estimates derived within subpopulationgrade 8 mathematics
7B Parameters and linked estimates derived within subpopulationgrade 8 science
8 Comparison of the component of variance due to model misspecification estimated by with its expected value estimated by for grade 8
9 Value of the component of due to temporal shift for grade 8
10 Total variance of with percentages due to components for grade 8
11 Parameter estimates for the linking of public school NAEP to TIMSS for grade 8
12 Components of for the public school link for grade 8
13 International marker levels of achievement for grade 8
14 Predicted NAEP cutpoints and their standard errors corresponding to the TIMSS marker levelspublic school linking for grade 8
15 Comparison of actual TIMSS mean proficiency with predicted TIMSS proficiency from NAEP results (data are from public schools only) for grade 8
16 Ninety-five percent confidence intervals for the percentages above the TIMSS marker levels based on actual TIMSS data and on predictions from NAEP (data are from public schools only) for grade 8
17 Estimated TIMSS scores from public school, 1996 NAEP data for states and jurisdictions: Grade 8 mathematics and science
18 Actual 1995 TIMSS scores for countries: Grade 8 mathematics and science
19 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in mathematics at grade 8 based on estimates from 1996 NAEP public school data for states and jurisdictions
20 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in science at grade 8 based on estimates from 1996 NAEP public school data for states and jurisdictions
21 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in mathematics at grade 8 based on actual 1995 TIMSS data for countries
22 Ninety-five percent confidence intervals and estimates for percent of students reaching TIMSS International Marker Levels in science at grade 8 based on actual 1995 TIMSS data for countries
List of Figures
1 Marginal distributions for pairs of hypothetical tests
2 Scatter plots of the hypothetical test score pairs
3 Rootograms comparing proficiency distributions for 1995 TIMSS and 1996 NAEP (NAEP distributions adjusted to have same mean and standard deviation as TIMSS)