Structured Abstract
Research design and methods
Data collection for the systematic review will go through three stages: literature search, article screening, and coding.
For the literature search, the team will conduct electronic bibliographic searches from the following databases: Academic Search Premier, Education Source, ERIC, and ProQuest Dissertation and Theses. The team chose these databases because they are known to include a high volume of educational research. To query these databases, they developed a Boolean expression for search that will capture a comprehensive set of articles using propensity scores with multilevel data in education. Variations of this expression will be pre-tested, and the research results will be compared to validate its effectiveness. In the Boolean expression, terms will be connected by AND/OR and the asterisk will be used as a wildcard to allow variations of names. The research team will include 20 years of education research in the systematic review by setting the date range from 2005 to 2025. They will still start in 2005 because the earliest studies about PSA with multilevel data in education are after 2005. They will search for articles in English published in peer-reviewed articles, theses, and dissertations. The terms of interest will be searched in the title, keywords, and abstract. The identified references will be downloaded using a RIS format, a tagged format for bibliographic information.
The study screening process will be carried out using the Covidence.org platform, a widely recognized tool developed by the Cochrane group for systematic reviews in the health sciences. The study screening will involve a two-stage process, consisting of abstract screening and full-text screening. Two trained reviewers will independently assess each article in both stages. To ensure consistency and reliability, reviewers will undergo training, which includes independently rating a sample of 10 abstracts, followed by thorough discussion and analysis of any discrepancies until a consensus is reached. Each reviewer will provide an "include" or "exclude" recommendation, and in cases of disagreement, Covidence will involve a third reviewer to evaluate the study. All articles that successfully pass the two-stage screening will enter the coding phase. The coding form will be created using the MUTOS framework for systematic review (Becker, 2017). MUTOS is an acronym for the components of the study that are of interest in a methodological systematic review: Methods (M), Units (U), Treatments (T), Observing operations (O), Setting (S). By mapping one or multiple fields of the form to each of the components of the MUTOS framework, we will ensure that our study collects comprehensive data about the methodological aspects of the studies.
Once the final version of the data extraction form is complete, the teamwill follow a robust process for reliable coding where all articles are independently coded by two reviewers, and disagreements are marked so that a third reviewer can check them and obtain consensus.
User Testing: User experience testing of the project’s products will be performed with 10 Ph.D. students from a variety of doctoral programs in education. User experience testing will be completed by asking each doctoral student to review the products and complete a user experience survey, then performing debriefing interviews about their responses to the survey.
Data analytic strategy
The data analysis for the systematic review will consist of organizing the practices identified in the review according to the six steps of propensity score analysis: 1) data preparation, 2) propensity score estimation, 3) propensity score method implementation, 4) covariate balance evaluation, 5) treatment effect estimation, and 6) sensitivity analysis. For each step, we will enumerate the specific context that methodological choices were made in the studies reviewed (e.g., level of treatment, distribution of numbers of clusters and cluster sizes), calculate the frequency of each method used (e.g., models to estimate propensity scores, weighting, matching, stratification, treatment effect estimators), and describe how methods used align with the results of existing methodological work on PSA of multilevel data.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
The project will produce a systematic review paper, training courses about multilevel PSA offered at academic conferences, flowcharts and checklists at the project’s website, and a YouTube video series.
Use in Applied Education Research: The systematic review will serve as a comprehensive guide for estimating treatment effects through PSA with multilevel data. For novice applied researchers, the products of the review will provide a structured and accessible roadmap, meticulously breaking down each step of PSA with multilevel data. Through the identification of common practices and their alignment with existing methodological literature, the review will establish a solid foundation upon which researchers can construct a nuanced understanding of PSA. This, in turn, will not only empower them to confidently navigate the complexities of PSA with multilevel data but also will instill the confidence to embark on methodologically sound investigations, thereby elevating the quality and rigor of educational research. In the case of experienced researchers already adept in PSA, the systematic review functions as a refined tool for decisionmaking. By offering nuanced support tailored to specific data situations, such as guiding the selection of the most appropriate method given specific data and study conditions and identifying accessible resources, the review facilitates efficient and informed decision-making. This tailored support not only streamlines the educational research process but also contributes to the optimization of resources, ensuring that experienced researchers can employ PSA methodologies with a heightened level of precision. Moreover, the systematic review will not only provide practical guidance for multilevel PSA, but also contribute to future research by shedding light on gaps and uncertainties within the current literature. By identifying these knowledge voids, the review serves as a catalyst, urging scholars to delve deeper into specific aspects of PSA with multilevel data. This proactive approach towards pinpointing research needs not only enriches academic discourse but also cultivates a responsive and evolving methodological environment. For methodologists aiming to evaluate PSA methods through Monte Carlo simulations, the review will serve as a guide to which conditions are realistic in applications of PSA to multilevel educational data, thus enabling them to create simulations with stronger generalizability to real data.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
