Project Activities
The team aims to provide resources for researchers and practitioners to (a) implement MI testing in accordance with current best practices (including interpretation of results), (b) make defensible and evidence-based decisions with respect to handling any measurement noninvariance revealed during analyses, (c) understand the potential consequences and implications of such decisions for the use of their focal instruments, and (d) improve reporting practices by thoroughly documenting all related analytical decisions and results. They expect our toolkit to contribute to raising awareness of the importance of testing for MI and increase the extent to which researchers implement the practice in their studies as a preliminary step to conducting group comparisons, examining change over time, and making selection decisions. The team also expects this increased awareness to be reflected in the extent to which journal and grant reviewers request that authors conduct and report on measurement equivalence testing in their manuscripts.
Structured Abstract
The research team will engage in three phases of work to develop the proposed MI toolkit: review of the existing literature, development of toolkit materials (e.g., example code, instructional text) and an accompanying website, and end-user testing for usability and feasibility.
The literature review will capture all relevant advances informing current best practices regarding MI testing.
The toolkit will cover foundational knowledge of the method, including conducting confirmatory factor analysis as a preliminary step to testing for MI, as well as guidance on the decisions needed at each step of the MI testing process (i.e., initial data exploration, conducting the analysis, interpreting results, handling noninvariant items, and reporting results). The toolkit will include example materials for several MI models.
End-user testing will engage participants in using example materials for the different MI models and providing detailed feedback on their usability and feasibility.
Research design and methods
User Research: Before releasing the materials and website publicly, the team will host them on a password-protected site for the purpose of conducting usability and feasibility testing with potential users. They will recruit a total of 30 end users, who will be assigned to one of the MI methods covered in the toolkit. Users will test the materials over three stages. In the first stage (Prep) lasting approximately 60 minutes, users will review materials for the model they have been assigned and conduct MI analysis. In the Testing stage, a 45-minute virtual session, users will provide feedback on issues such as the ease of locating materials on the website, the clarity of materials (e.g., conceptual explanations, explanations of code and output, reporting templates), the usefulness of materials, and recommendations to improve the clarity and helpfulness of the toolkit. Approximately 1 week after the user testing, in the final stage, the research team will conduct a 30-minute semi structured debrief interview with each tester. All test users will be compensated for their time and effort.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The American Institutes for Research® (AIR®) will develop a web-based toolkit comprising five key components:
- a discussion of the importance of examining MI,
- an overview of MI,
- step-by-step tutorials (annotated code and output files for conducting analyses and handling any noninvariance revealed),
- sample data sets to help users gain hands-on experience estimating MI models, and
- guidelines and examples for reporting the results of MI analyses.
All toolkit materials will be made freely available on a website hosted by AIR. They will disseminate the toolkit through conference workshops, webinars, and AIR’s social media accounts. Specifically, conferences such as the American Educational Research Association and Society for Research in Child Development offer ideal venues in which to share this work because their attendees closely match our intended audience: researchers and graduate students with varied levels of methodological expertise whose work involves comparing outcomes or other scale scores across groups or over time. They will also develop and conduct one or more webinars, which will be posted to the toolkit’s website and on the PI’s and AIR’s social media accounts (e.g., Twitter/X, LinkedIn), including one or more brief descriptive flyers to encourage use of the MI toolkit by practitioners. The team will also write and publish several blog posts, sharing them on social media and making use of the extensive professional networks maintained by the PI, co-PI, our external advisors, and AIR to disseminate knowledge and encourage adoption of the MI toolkit.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
