For at least two decades, both the public and private sectors have called for increased federal investment in computer science (CS) education to better prepare students for new and emerging careers in national defense, cybersecurity, and a global economy. Despite well-founded fears that artificial intelligence (AI) and automation are disrupting career pathways, U.S. Department of Labor projections suggest that the need for computer-related talent in the United States persists.
K–12 schools are struggling to give students a strong foundation for pursuing careers in computer science. According to a recent Code.org study, only:
- 19% of high school seniors graduating in 2024 took at least one high school computer science class, and
- 12 states have implemented CS graduation requirements.
Education leaders across the country are implementing a variety of strategies to address this challenge. However, to guide and monitor these efforts, decisionmakers need clear, reliable data and information about whether and how CS curricula and programs are achieving intended results—such as building students’ computing skills and interest in computer science. This evidence can guide program improvement, help sustain funding and support, and inform efforts to scale and replicate effective CS interventions.
Evidence of what works is lacking. Few evaluations of CS interventions meet one of the top three evidence tiers under the Every Student Succeeds Act (ESSA). Gaps in high-quality research on the effectiveness of computer science programs limit leaders’ ability to make evidence-informed decisions about which curricula or programs to adopt and why.
Building evidence on the effectiveness of CS interventions is challenging. With few standardized measures of computational thinking, coding, and other computing skills, it’s difficult for educators to credibly evaluate whether a specific CS course, pathway, or intervention is improving student learning and outcomes overtime (such as increasing the number of students who earn CS-related dual enrollment credits). Local school systems rarely have the resources, expertise, and capacity to develop and validate their own assessments, making it difficult for state and local education agencies to collect and analyze credible data on student performance in CS, identify how programs are influencing student outcomes, and confidently make data-informed instructional and professional development decisions.
A practical way forward for education leaders
REL Mid-Atlantic is partnering with the Maryland Center for Computing Education (MCCE) and St. Mary’s County Public Schools (SMCPS) to build their local capacity to design and conduct studies of CS interventions that can meet evidence standards under the federal Every Student Succeeds Act. Beginning in the 2019/20 school year, SMCPS split a double math period into one period of math and one period of computer science in response to Maryland’s Securing the Future: Computer Science for All Act (HB0281), which encourages local school systems to introduce CS learning in middle and elementary grades, and Maryland’s state ESSA plan that required adding computational thinking as a component of a well-rounded middle school curriculum. SMCPS’s schedule change not only enabled all eighth grade students to take a required introductory computer science course, Code.org’s Computer Science Discoveries (CSD), but also created a pathway for middle school students to meet the state’s Computer Science, Engineering, or Technology Education graduation requirement (COMAR 13A.04.01.01) before entering high school.
MCCE, SMCPS and REL Mid-Atlantic are investigating whether splitting a double math period negatively impacted mathematics achievement, and whether taking CSD increases student interest and success in advanced CS coursework.
Below, we describe three research design activities used to plan our study. The materials and examples can support other education agencies and research-to-practice partnerships in designing rigorous evaluations of CS interventions. We completed these activities iteratively and cyclically over a three-month planning process, rather than in sequence.
Articulate the relationship between a CS program’s intended design and desired short- and long-term outcomes.
Clearly articulating a program’s key components helps establish a shared understanding of how it is intended to work and guides the study of its implementation. If components are not implemented as intended, the program may not achieve desired results. Leaders can use a logic model to document and describe core resources, activities, and measurable outputs that indicate whether components were carried out as intended, and therefore, likely to achieve the program’s desired outcomes.
The SMCPS logic model example provides a structure that education leaders can use to map out key program components (including professional learning and instructional resources provided to CS teachers), outputs that should be anticipated if a program has been implemented as designed, short-term or leading indicators of whether the program is on track to produce desired outcomes, and desired long-term outcomes. We used our logic model as a living document that we updated and refined based on the quality and availability of program and student performance data needed to measure identified outputs and outcomes.
Align research questions with S.M.A.R.T. outcomes.
Throughout our planning process, we considered whether our desired outcomes are S.M.A.R.T.—specific, measurable, achievable, relevant, and time-bound—using this evidentiary support template, which also includes an example from SMCPS, to:
- Review and revise a list of desired outcomes to ensure each was specific, measurable, and time-bound (such as by specifying the math and CS courses and school years the study would investigate).
- Summarize research on similar CS interventions to determine whether our outcomes were relevant and achievable. To narrow our desired outcomes list, we only retained those with emerging or demonstrated evidence of a positive relationship between our program components and student math or CS outcomes.
We then refined our research questions to ensure they aligned with our narrowed list of S.M.A.R.T. outcomes and updated our logic model accordingly.
Identify appropriate and feasible data sources and methods.
Sometimes research questions cannot be answered because of data collection limitations, data quality issues, or analytic constraints. Answering our research questions requires comparing the longitudinal CS and math performance of SMCPS eighth graders who took the required CSD course to eighth grade students in other Maryland local school systems who did not take it. A key barrier associated with designing this study has been a lack of summative, standardized CS course performance data at the state and local levels. These data are needed to compare short- and long-term achievement outcomes between CSD and non-CSD course takers. With the exception of the College Board’s AP computer science course exams, SMCPS did not have access to assessments of student learning in CS that
- are appropriate for use in a K-12 setting,
- align with national or their state K-12 CS learning standards,
- align with the content of SMCPS’s CS courses,
- provide performance results that indicate a student’s readiness for more advanced CS coursework, or
- are administered statewide as an end-of-course exam.
To address this barrier, we used this data and methods mapping template, which also includes an example from SMCPS, to identify and assess the fit of available data sources and analytic approaches for reliably answering our research questions. When data were limited, we refined or removed questions or chose less rigorous analyses (such as correlation instead of regression). For example, we had originally hoped to evaluate whether CSD students were more likely than non-CSD students to enroll and succeed in high school CS and career and technical education courses. Because Maryland doesn’t administer standardized end-of-course exams for these courses, we refined one of our analyses to focus solely on CS and CTE course enrollment and another to focus solely on AP CS course and exam performance.
Up next
We’re working with MCCE and SMCPS to conduct the study designed through this process, with results expected later this year. We hope to share findings that can help other school systems explore creative ways of adjusting school schedules to increase access and opportunity for CS coursework without adversely affecting student achievement in other core courses. To stay informed about study findings and related resources, follow the Institute of Education Sciences on social media and subscribe to the REL Mid-Atlantic newsletter.
