​​​NAEP Technical DocumentationDevelopment of the 2019 Grade 4 Science Indices

For the 2019 grade 4 science assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2019 grade 4 science indices can be summarized in three main steps: 

1. Question development. New sets of contextual items, such as those exploring students’ persistence in learning and their enjoyment of complex problems, were developed and included in the science student questionnaire. Through content analysis as part of the item development process, only sets of items that were theoretically interpretable and meaningful as a conceptual unit were included as potential indices to measure specific constructs of interest. 

2. Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.   

3. Index scoring. The partial credit Item Response Theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on marginal maximum likelihood methodologies. After the parameters were estimated, expected a posteriori (EAP) scores were calculated as the estimate of the index score. When a student does not provide a valid response to a contextual item, the student will not receive an index score for the index containing that item. ​In the first administration of an index, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2 on a scale from 0–20. In subsequent administrations, EAP scores for established (i.e., trend) indices are estimated using concurrent calibration together with data from the previous year and then transformed using linear transformation (Kolen and Brennan 2004​) to be on the same scales created in the initial administration. Transformation constants ​ and are calculated as follows:

where and are the target standard deviation and mean of the transformed scores from the previous year, and and are the provisional standard deviation and mean of the previous year EAP scores in the ​current year concurrent calibration.

Index of Students' Persistence in Learning

The table below presents the items forming the index of students' persistence in learning. This index was designed to measure students' tendency to persevere and work hard in the face of challenges. Grade 4 students were asked to indicate how much each of the four statement items described them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Index of Students' Enjoyment of Complex Problems

The table below presents the items forming the index of students' enjoyment of complex problems. This index was designed to measure students' enjoyment of problems and activities that challenge them to think. Grade 4 students were asked to indicate how much each of the four statement items described them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Index of Students' Academic Self-Discipline

The table below presents the items forming the index of students' academic self-discipline. This index was designed to measure students' tendency to pay attention and stay on task when learning. Grade 4 students were asked to indicate how often they have done each of the four statement items (never or hardly ever, less than half of the time, about half of the time, more than half of the time, or all or almost all of the time).

Index of Students’ Confidence in Science Knowledge and Skills

The table below presents the items forming the index of students' confidence in their science knowledge and skills at grade 4. This index was designed to measure students' belief in their abilities to do various science-related tasks. Grade 4 students were asked to indicate their confidence in doing the task described in each of six items (I definitely can’t, I probably can’t, maybe, I probably can, or I definitely can).

Index of Students’ Performance Goals in Science

The table below presents the items forming the index of students' performance goals in science. This index was designed to measure students' motivation to show others that they are good at science and get better grades than other students in their science class. Grade 4 students were asked to indicate how much each of the four statement items described them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or exactly like me).

Printer-friendly Version