Project Activities
The AmplifyGAIN Center will: (1) conduct exploratory studies to examine how GenAI is currently used in classrooms to improve teaching and learning outcomes, (2) informed by the exploratory studies, AmplifyGAIN will develop, test, and revise "Colleague AI" — a GenAI assistant for teachers to prepare rigorous, engaging, and inclusive math and science lesson materials, conduct formative classroom assessments, and automatically score and generate diagnostic reports to personalize student feedback; and (3) conduct a pilot study to assess the promise of Colleague AI for improving learners' math and science education outcomes. In addition, AmplifyGAIN will provide national leadership and outreach activities on the responsible use of GenAI to improve learner outcomes.
Focused program of research
The AmplifyGAIN Center will conduct three intertwined areas of work. First, two types of exploratory studies will be conducted: in-depth, classroom-based case studies and national longitudinal surveys of teachers to examine how AI is used in classrooms. Findings from the exploratory studies will be used to refine theories of change about how GenAI tools can transform math and science teaching and learning and will inform the development and refinement of Colleague AI. Second, AmplifyGAIN will iteratively develop and refine Colleague AI and conduct two phases of usability studies to identify bugs and gather human feedback and user activities to iteratively improve the tool. Third, AmplifyGAIN will conduct a rigorous pilot test of Colleague AI in Washington state school districts to assess its promise for improving teacher and learning outcomes in math and science.
National leadership and outreach activities
The AmplifyGAIN Center will provide national leadership on the use of GenAI in mathematics and science education through partnerships with other research institutions, K–12 school systems, technology industries, private funding firms, and advocacy groups. National leadership, capacity building, and outreach activities will include technical training on the research and use of GenAI in education to researchers, data scientists, and software engineers, as well as professional development materials for K–12 teachers and school technology directors, the creation of online community forums, webinars, and policy briefs for key stakeholders.
Structured Abstract
Setting
K–12 teachers across the United States will be included in the national survey of teachers and in the usability study. Classroom-based exploratory studies and the pilot study will take place in K–12 schools in Washington state.
Sample
The classroom-based exploratory studies will include a diverse group of 30 focal teachers who vary on a number of characteristics including grade/subject (K-5, 9-12 for math and K-12 for science), school demographics (e.g., district financial status, urbanity, percentages of students eligible for free- and- reduced price lunch, special education accommodations, and multilingual programs), as well as teachers' and their students' race/ethnicity. In addition, a nationally representative sample of 2,000 K-12 math and science teachers will be surveyed. The research and development of Colleague AI will begin with 10 teacher participants (5 elementary and high school math teachers and 5 science teachers), followed by a larger sample of at least 2,000 teachers (elementary and high school math and K-12 science teachers) for usability testing. The pilot study will include 420 math and science teachers in grades 3-10 from a total of 42 schools (20 elementary schools; 22 secondary schools), with approximately 10 teachers from each school. The Center will recruit teachers for the pilot study from school districts that serve students with diverse needs in language, cultural, special education accommodations, academic mastery levels, and from urban, suburban, small town, and rural communities.
Research design and methods
The exploratory studies will include in-depth classroom-based case studies with a set of focal teachers from diverse backgrounds and other educators and parents in their professional networks. Exploratory studies will also include longitudinal, nationally representative teacher surveys to gauge the trends of how generative AI tools transform teaching and learning. Using an Agile Technology Development framework, the research team will be able to conduct rapid analysis of exploratory study data to inform the understanding of AI-enhanced math and science instruction and inform the iterative development of Colleague AI.
Colleague AI's usability testing during the product development phase involves two sequential phases: a testing phase with a small group of teachers that uses visual tracking and cursor tracking of teachers' attention for information, activity history data, and teacher interviews. The second stage of usability testing invites all eligible teachers nationwide and uses A/B testing to assess the interface design, algorithm functionality, and system performance. Usability testing will examine which AI model or product feature is more useful to assist with which instructional task, and study how teacher backgrounds affect their use of the Colleague AI features. The analysis will be conducted in a timely manner to support rapid cycles of product improvement.
After all proposed features are developed and tested, the AmplifyGAIN Center will conduct a pilot study, using a mixed methods approach with a randomized controlled trial. They will examine the relationship between teachers' use of Colleague AI, their lesson plan quality and instructional effectiveness, and their students' math and science learning outcomes. The research team will randomly assign schools to either the treatment condition to receive the Colleague AI tool and relevant support to ensure teachers' active engagement, while schools in the control conditions will experience business as usual. Relevant supports for treatment group teachers include, but are not limited to, webinars, tutorials and FAQ page, on-demand technical assistance, online forum, and in-person professional learning sessions.
Key measures
Key teacher outcome measures include teachers' lesson plan quality coded by a combination of human experts and computer algorithms based on validated rubrics, self-reported workload stress, structured self-reflection on instructional effectiveness, and classroom observations guided by well-establish observation rubrics in math (MQI or IQA) and science (EQuIP). In both treatment and control groups, student outcome data include students' engagement in math and science classes, formative assessment scores, and math and science proficiency level on state standardized tests. Lastly, the Center will collect program implementation data through Colleague AI's platform activity history data, surveys of teachers' frequency and depth of generative AI use in instruction, training, and school supports for lesson preparation and instruction.
Data analytic strategy
For the exploratory studies and usability testing, iterative descriptive analyses of qualitative and quantitative data will inform teacher training material development and Colleague AI's product development. During the pilot study, the AmplifyGAIN Center will estimate both the intent-to-treat and treatment-on-the-treated effects using a reduced-form and an instrumental variable approach. They will analyze detailed, multidimensional implementation measures to understand the mechanisms of change and explain possible heterogeneous effects varying by teacher experience, school contexts, and classroom student compositions. The findings will inform both national leadership activities and Colleague AI's continuous integration and improvement.
Cost analysis strategy
The Center will use the ingredients methods to keep a detailed record of the costs and use cost-effectiveness ratios to compare Colleague AI with other AI-powered tools or instructional technology interventions reported in published studies.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: The project findings will provide significant contributions to foundational knowledge in multiple disciplinary areas, including AI use theories that describe the relationships among technology, students, content, and teachers in dynamic classroom settings; design principles of AI technologies for education; fine-tuning AI algorithms that integrate with domain knowledge; and advancing formative assessments and personalized feedback to students. In addition, the AmplifyGAIN Center will complete the development of Colleague AI, which will initially be offered to all teachers nationwide for free in math and science. The Center will produce publications and presentations and other dissemination products (for example, website, webinar) that will reach educators, school technology specialists, administrators and policymakers, researchers, and EdTech developers.
ERIC Citations: Find available citations in ERIC for this award here.
Related projects
Supplemental information
Co-Principal Investigators: He, Jian; Shapiro, R. Benjamin; Wang, Chun; Edwards, Ann; Nucci, Drew; Sarkar, Shawon
Partner Institutions:Hensun Innovations, Inc.; WestEd; Washington Office of Superintendent of Public Instruction
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
