TIMSS and TIMSS Advanced are an international collaborative effort involving representatives from every country participating in the studies. For TIMSS and TIMSS Advanced 2015, the test development effort began with a review and revision of the frameworks that are used to guide the construction of the assessment. The frameworks were updated to reflect changes in the curriculum and instruction of participating countries and education systems. U.S. and international experts in mathematics and science curriculum, education, and measurement, and representatives from national educational centers around the world contributed to the final content of the frameworks. Maintaining the ability to measure change over time was an important factor in revising the frameworks.8
Most assessment items are carried forward from previous TIMSS and TIMSS Advanced assessments in order to measure trend. National Research Coordinators (NRCs) and experts from each country, as well as the IEA Science and Mathematics Review Committee (SMIRC) contributed to the development of new assessment items. The SMIRC consisted of 16 content area specialists in mathematics or science from various countries. All new items were reviewed by NRCs, the SMIRC, and other subject-matter experts to ensure that the content, as explicated in the frameworks, was covered adequately. Items were field-tested in most of the participating countries. Results from the field test were used to evaluate item difficulty, how well items discriminated between high- and low-performing students, the effectiveness of distracters in multiple-choice items, scoring suitability and reliability for constructed-response items, and evidence of bias toward or against individual countries or in favor of boys or girls.
More detail on the distribution of new and trend items for TIMSS and TIMSS Advanced are included in the tables below.
Design of instruments
TIMSS and TIMSS Advanced 2015 included student booklets containing assessment items as well as non-cognitive questionnaires. Questionnaires for principals and teachers were self-administered, primarily using an online survey system.
The assessment booklets were constructed such that not all of the students responded to all of the items. This is consistent with other large-scale assessments, such as NAEP. To keep the testing burden to a minimum, and to ensure broad subject- matter coverage, TIMSS used a rotated block design that included both mathematics and science items for each student. That is, students encountered both mathematics and science items during the assessment. TIMSS Advanced also used a rotated block design, but each student took only one subject, either advanced mathematics or physics.
The fourth-grade assessment consisted of 14 booklets, each requiring approximately 72 minutes. The eighth-grade assessment also consisted of 14 booklets, each requiring approximately 90 minutes. The assessments were administered in two equally timed sections, with a 5- to 10-minute break in between. At both grades, the mathematics and science items were each assembled separately into 28 blocks of items. Booklets consisted of four blocks each, including at least one mathematics block and at least one science block.
The TIMSS Advanced assessments consisted of six booklets for advanced mathematics and six booklets for physics, each requiring approximately 90 minutes. Each student completed just one booklet. In both subjects, booklets consisted of three blocks each.
After the cognitive assessment, students completed a 30-minute questionnaire designed to provide information about their backgrounds, attitudes, and experiences in school.
TIMSS and TIMSS Advanced 2015 included questionnaires for principals, teachers, and students. Questionnaires were based on context questionnaire frameworks.9 Questions were developed in an international collaborative process involving NRCs, the TIMSS 2015 Questionnaire Item Review Committee (QIRC), and staff at the TIMSS & PIRLS International Study Center. The QIRC comprises seven policy analysis experts from different countries. Like the assessment items, all questionnaire items were field-tested and the results reviewed carefully. After the review, some of the questionnaire items were revised prior to their inclusion in the final questionnaires.
The questionnaires are designed to provide context for student achievement, focusing on such topics as students' attitudes and beliefs about learning, their habits and homework, and their lives both in and outside of school; teachers' attitudes and beliefs about teaching and learning, teaching assignments, class size and organization, instructional practices, and participation in professional development activities; and principals' viewpoints on policy and budget responsibilities, curriculum and instruction issues and student behavior, as well as descriptions of the organization of schools and courses.
Online versions of the school and teacher questionnaires were offered to respondents as the primary mode of data collection.
Source versions of all instruments (assessment booklets and questionnaires, and procedural manuals) were prepared in English by the TIMSS & PIRLS International Study Center and translated into the primary language or languages of instruction in each education system. In addition, it was sometimes necessary to adapt the instrument for cultural context, even in countries and education systems that use English as the primary language of instruction. All translations and adaptations were prepared under comprehensive guidelines established by the IEA Secretariat. Well-trained and experienced verifiers reviewed and documented the quality and comparability of national instruments to the international versions. The goal of the translation and adaptation process is to ensure that the meaning nor difficulty of items is changed.
Further details on the translation and adaptation process can be found at http://timssandpirls.bc.edu/publications/timss/2015-methods/T15_MP_Chap7_Translation_And_Verification.pdf for TIMSS and for TIMSS Advanced.
8 The complete subject area frameworks for TIMSS and TIMSS Advanced 2015 are available on the TIMSS international website at http://timssandpirls.bc.edu/timss2015/frameworks.html and http://timssandpirls.bc.edu/timss2015-advanced/frameworks.html, respectively.
9 The context questionnaire framework for TIMSS and TIMSS Advanced 2015 are available at: http://timssandpirls.bc.edu/timss2015/downloads/T15_FW_Chap3.pdf and for TIMSS Advanced at: http://timssandpirls.bc.edu/timss2015-advanced/downloads/TA15_FW_Chap3.pdf, respectively.