​​​​​​​​​​​​​​NAEP Technical DocumentationDevelopment of the Social Studies Indices

For the 2014 social studies assessments (civics, geography, and U.S. history), several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2014 social studies indices can be summarized in three main steps: 

1. Question selection. A content analysis of the existing 2014 social studies survey questionnaires was conducted to identify sets of questions or items that were theoretically interpretable and meaningful as a conceptual unit, as potential indices to measure specific constructs of interest.

2.  Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.   

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on marginal maximum likelihood methodologies. After the parameters were estimated, expected a posteriori (EAP) scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2 on a scale from 0–20.  

Indices of Students' Views of Civics, Geography, and U.S. History

The tables below present the items making up the indices of students' views of civics, geography, and U.S. history, respectively. The indices were designed to measure students' view of their social studies coursework based on their level of agreement with five items. For each of four statements, students were asked to indicate how often they agreed with the statement (never or hardly ever, sometimes, often, or always or almost always). A fifth item asked students to indicate the extent to which they agreed that social studies, civics, geography, or U.S. history was one of their favorite subjects (not at all, a little, or a lot). Statements expressing negative perceptions (e.g., "The U.S. history or social studies work is difficult.") were reverse coded during the scaling process. 
   

Top

The partial credit IRT model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on marginal maximum likelihood methodologies. The following tables show the IRT parameters for the index of students' views of civics, geography, and U.S. history, respectively.

Top

After the parameters were estimated, EAP scores were calculated as the estimate of the index score, and transformed to a reporting scale of 0–20.

As a reporting aid, index scores were divided into a range of categories or classifications (e.g., low, moderate, high). The cut points selected to divide the index scores into meaningful categories were based on the distributions of the response average of each index. For instance, for the index of students' views of civics, students were classified as follows:​

• Students with a response average less than 2 were classified as low on the index. That is, students who on average responded closest to never or hardly ever on a question with response options never or hardly ever, sometimes, often, and every day or almost every day generally reported that their course material was difficult and indicated little interest in the topics studied.
• Students with a response average greater than or equal to 2 but less than 3 were classified as moderate on the index.
• Finally, students with a response average greater than or equal to 3 were classified as high on the index. That is, students who on average responded closest to often generally reported that their course material was easy and expressed a strong interest in the topics studied in their courses.
  

The tables below show the ​transformed scale scores and score classifications ​for the civics, geography, and U.S. history indices, respectively.

Printer-friendly Version