​​​​​NAEP Technical Documentation Development of the 2017 Grade 4 Mathematics Indices

For the 2017 grade 4 mathematics assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2017 grade 4 mathematics indices can be summarized in three main steps:

1. Question development. New sets of contextual items, such as those exploring students’ persistence in learning and their enjoyment of complex problems, were developed and included in the mathematics student questionnaire. Through content analysis as part of the item development process, only sets of items that were theoretically interpretable and meaningful as a conceptual unit were included as potential indices to measure specific constructs of interest.

2. Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on marginal maximum likelihood methodologies. After the parameters were estimated, expected a posteriori (EAP) scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2 on a scale from 0–20.

Index of Students' Persistence in Learning

The table below presents the items forming the index of students' persistence in learning. This index was designed to measure students' tendency to persevere and work hard in the face of challenges. Grade 4 students were asked to indicate how much each of the four statement items described a person like them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Index of Students' Enjoyment of Complex Problems

The table below presents the items forming the index of students' enjoyment of complex problems. This index was designed to measure students' enjoyment of problems and activities that challenge them to think. Grade 4 students were asked to indicate how much each of the four statement items described a person like them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Index of Students’ Confidence in Mathematics Knowledge and Skills

The table below presents the items forming the index of students' confidence in their mathematics knowledge and skills at grade 4. This index was designed to measure students' belief in their abilities to do various mathematics-related tasks. Grade 4 students were asked to indicate their confidence in doing the task described in each of four items (I definitely can’t, I probably can’t, maybe, I probably can, or I definitely can).

For each index item, the response categories were scored as numerical values (e.g., for an item with five response categories, category A was scored as 1, B was scored as 2, C was scored as 3, D was scored as 4, and E was scored as 5). For the 2017 grade 4 mathematics indices, item response categories were collapsed; scores for a five-category item thus became 1, 2, 3, and 4 after collapsing. The table below describes the treatment of the index items.

The partial credit IRT model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on marginal maximum likelihood methodologies. The following tables show the IRT parameters for the 2017 grade 4 mathematics indices.

After the parameters were estimated, EAP scores were calculated as the estimate of the index score, and transformed to a reporting scale of 0–20. The tables below show the response averages and transformed scale scores for each of the three grade 4 mathematics indices. Note that response averages represent the average scored responses after score collapsing (if there was any collapsing)​. Each response average corresponds to one transformed score. The increment used for increases in the response average is determined by the number of items that form an index. For example, an index with four items would have response averages that increase by an increment of 1 / 4 or 0.25, while an index with five items would have response averages that increase by an increment of 1 / 5 or 0.20.

As a reporting aid, index scores were divided into a range of categories or classifications (e.g., low, moderate, high). The cut points selected to divide the index scores into meaningful categories were based on the distributions of the response average of each index.

As an example, for the index of persistence in learning, grade 4 students were classified as follows:

• Students with a response average less than 2 were classified as low on the index. That is, students who on average responded closest to not at all like me on a question with the five response options not at all like me, a little bit like me, somewhat like me, quite a bit like me, and very much like me were classified as having a low level of persistence in learning.
• Students with a response average greater than or equal to 2 to less than 3 were classified as having a moderate level of persistence in learning.
• Finally, students with a response average greater than or equal to 3 were classified as high on the index. That is, students who on average responded closest to very much like me were classified as having a high level of persistence in learning.

Printer-friendly Version