Inside IES Research

Notes from NCER & NCSER

New Project Exploring Adult Basic Skills in STEM-Related Postsecondary CTE®

In celebration of CTE® (career and technical education) month, we would like to highlight the launch of an NCER project that aims to help us understand how to best support adults seeking additional CTE education and training.

Through their exploratory project, Literacy, Numeracy, and Problem-Solving Skills in Technology-Rich Environment in the STEM-Related Subbaccalaureate Programs in the United States, researchers will use a mixed-method design to gather information about the distribution of basic skills (literacy, numeracy, and problem solving) for adults in STEM occupations and students enrolled in STEM-related sub-baccalaureate programs at community colleges. Their goal is to help identify the needs of students and the programming practices at community colleges that may promote basic skill development in STEM programs.

The team will be leveraging data from the Program for the International Assessment of Adult Competencies (PIAAC) to survey the distribution of skills and abilities across a nationally representative sample of adults in STEM fields. They will also be collecting primary data from adults and programs in multiple locations, including Indiana, Ohio, and Washington states.

 

 

To help inform the public about their project, the researchers have created a short YouTube video for the public. This project began in July 2020 and may have initial results ready as early as late 2021.

 


Written by Meredith Larson (Meredith.Larson@ed.gov), Program Officer for Postsecondary and Adult Education, NCER.

Facilitating Causal Research in CTE: Notes from the Network

The Career and Technical Education (CTE) Research Network just entered its third year, and it is time to share recent accomplishments with the IES community! As a reminder, the CTE Research Network (led by the American Institutes for Research, or AIR) was created to expand the evidence base on the impact of CTE programs on student outcomes using causal research methods.

 

Research

In June 2020, AIR released a preliminary report of CTE sites that are ready for causal evaluation. Designed to support researchers interested in studying the impact of CTE on student outcomes, the report details the history, theory of change, student enrollments, and other information for 4 selected CTE sites around the country. The Network hopes that researchers will use the information in the report to design evaluation studies of these programs.

In July 2020, a fifth research project joined the network. A team from MDRC, led by Rachel Rosen, was recently funded by IES to study the Impact of Technology-Based Career Advising Tools on High School Students' CTE Choices and Academic Performance. In partnership with Communities in Schools (CIS), the study will use a three-arm, school-level random assignment research design (RCT) to assess the effects of Navience and YouScience on students' self-expressed attitude and interest in career pathways, CTE course taking patterns, and engagement with and progress towards graduation. We welcome the team to the Network and look forward to learning whether and how these advising tools influence student thinking about career options, choice of relevant CTE coursework and work-based learning options, and decisions about CTE concentration in available pathways and programs of study.

A small group of researchers from different Network teams collaborated on and recently released a technical working paper on counterfactuals in CTE. It can be challenging to identify comparison groups for CTE students because it is an elective into which they self-select. The paper describes a variety of rigorous methods of comparing CTE students to valid counterparts and provides case studies that illustrate how to use these methods.

 

Training

The CTE Research Network is committed to increasing the number of researchers trained to study CTE using causal methods. It is notoriously challenging to isolate the effects of CTE from other influences on student outcomes. In August 2020, the Network hosted 18 researchers for a week-long virtual summer training institute on how to design a causal study to examine the impact of CTE. During the week, participants learned how to implement randomized-control trials (RCTs), regression discontinuity designs (RDDs), and comparative interrupted time series (CITS) in a CTE context. After learning about each method, participants worked in small groups to apply the method to real data and had access to the instructors to ask questions. The feedback about the training was overwhelmingly positive. The lecture portions of the training will be posted soon to the training page of the Network’s website. Another week-long training institute will be held in summer 2021 (hopefully, in person!)

The Network is currently developing a series of online modules for CTE practitioners and state agency staff to strengthen capacity to access, conduct, understand, and use CTE research. There will be a presentation to preview the modules at the Association of Career and Technical Education (ACTE) Best Practices and Innovations Conference on October 9, 2020 and a longer and more in-depth session at ACTE’s Career Tech VISION conference the first week of December. These practitioner training modules will be available for free on the website in late Fall 2020.

 

Leadership and Dissemination

The CTE Research Network is regularly updating its resources page with publications of interest to the CTE research field. The most recent is a report of findings from MDRC’s study of P-Tech high schools. The Network’s equity workgroup (a group of researchers from across the Network’s member projects) also published a popular blog this summer on applying an equity lens to CTE research. The Network also posts outside resources such as a REL self-study tool on career readiness and evaluation reports from other researchers.

 


For more about the CTE Research Network, you can sign up to receive the Network’s quarterly newsletter at the bottom of their website’s home page and follow them on Twitter (@CTEResNetwork) and LinkedIn.

If you are interested in learning more about the CTE Research Network, contact the Director, Kathy Hughes (khughes@air.org).

If you are interested in discussing CTE research opportunities at IES, contact Corinne Alfeld (Corinne.Alfeld@ed.gov).

Cost Analysis in Practice (CAP) Project Provides Guidance and Assistance

In 2020, as part of a wider IES investment in resources around cost, IES funded the Cost Analysis in Practice (CAP) Project, a 3-year initiative to support researchers and practitioners who are planning or conducting a cost analysis of educational programs and practices. The CAP Project Help Desk provides free on-demand tools, guidance, and technical assistance, such as support with a cost analysis plan for a grant proposal. After inquiries are submitted to the Help Desk, a member of the CAP Project Team reaches out within two business days. Below is a list of resources that you can access to get more information about cost analysis.

 

STAGES FOR CONDUCTING A COST ANALYSIS

 

CAP Project Resources

Cost Analysis Standards and Guidelines 1.0: Practical guidelines for designing and executing cost analyses of educational programs.

IES 2021 RFAs Cost Analysis Requirements: Chart summarizing the CAP Project’s interpretation of the IES 2021 RFAs cost analysis requirements.

Cost Analysis Plan Checklist: Checklist for comprehensive cost analysis plans of educational programs and interventions.

Introduction to Cost Analysis: Video (17 mins).

 

General Cost Analysis Resources

The Critical Importance of Costs for Education Decisions: Background on cost analysis methods and applications.

Cost Analysis: A Starter Kit: An introduction to cost analysis concepts and steps.

CostOut®: Free IES-funded software to facilitate calculation of costs once you have your ingredients list, includes database of prices.

DecisionMaker®: Free software to facilitate evidence-based decision- making using a cost-utility framework.

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research: Open access journal article.

 

*More resources available here.


The content for this blog has been adapted from the Cost Analysis in Practice Project informational flyer (CAP Project, 2020) provided by the CAP Project Team. To contact the CAP Help Desk for assistance, please go to https://capproject.org/. You can also find them on Twitter @The_CAP_Project.

CTE Research Network Identifies Four Sites Ready to be Evaluated

In 2018, the IES awarded a grant1 to the American Institutes for Research (AIR) to lead a research network focused on career and technical education (CTE), the Expanding the Evidence Base for Career and Technical Education Network (CTE Research Network). The mission of the CTE Research Network is to increase the number of CTE impact studies and strengthen the capacity of the field to conduct and use rigorous CTE research.

One of AIR’s primary tasks as the CTE Network’s Lead is to conduct an evaluability study (also called a feasibility study) to identify CTE models or programs that could be evaluated using a rigorous experimental design. The purpose of the study is to ease the way for other researchers to evaluate CTE by doing the advance work to find suitable sites that may be interested in participating in research. Any interested research team may approach one of these sites to partner in an evaluation. IES and AIR hope that qualified teams will submit an application to the IES Education Research Grants program, under the CTE topic, for grant funding to conduct an evaluation.

In a preliminary report released today, the CTE Network Lead describes the method they used to identify a broad range of programs and models, the vetting criteria, and the reasons for selecting the four sites. For each of the selected sites, the report also describes the scope of the program and student enrollment, the CTE programs offered, the data available, and the willingness of the sites to welcome researchers to evaluate the CTE program. In addition, the report includes the suggested next steps for researchers and possible limitations in carrying out an evaluation of the particular site or program model.

Prior research on CTE over the last half century has mostly been exploratory in nature or, at best, quasi-experimental. One of the primary reasons for the lack of experimental research is that it is difficult to assign students to elective courses. Even quasi-experimental designs are challenging, as it is difficult to statistically control for all the reasons a student might choose to enroll in CTE. See here and here for further discussion of the challenges in conducting CTE research.

The CTE Research Network has another upcoming effort to help increase the CTE evidence base: a free training on causal methods for CTE research. The training will take place online in August 2020; the deadline for applications is June 30, 2020.

News about the CTE Research Network and resources to help CTE researchers can be found on the Network’s website; IES also occasionally blogs about the research findings of Network members. Although most of the CTE Network members are currently studying CTE at the secondary level, we hope that more research will be conducted at the postsecondary level. Researchers interested in applying to IES for a grant to study CTE are welcome to contact Corinne Alfeld (contact information below).


1Using Perkins funds from the Office of Career, Technical, and Adult Education (OCTAE) in partial fulfillment of the legislative requirement for a national research center to carry out scientifically-based research and evaluation for the purpose of developing, improving, and identifying the most successful methods for addressing the education, employment, and training needs of career and technical education (CTE) participants in CTE programs [Sec. 114(d)(4)].

 

Written by Corinne Alfeld (corinne.alfeld@ed.org), IES program officer, and Katherine Hughes (khughes@air.org), principal investigator for the CTE Network Lead at AIR

 

IES is Providing Digital Technical Assistance for FY 2021 Research Grant Applicants

Given the many challenges that this year has brought, including the difficulties and uncertainties due to the COVID-19 pandemic, IES is providing different resources and options to assist applicants as they begin preparing their applications. To ensure that program officers can focus their time on project-specific questions, applicants should review these resources first before seeking individual feedback.

First, have a copy of the documents that are needed to submit a proposal. Download a copy of the relevant request for applications (RFA) and the IES Application Submission Guide. This page has PDFs of these documents: https://ies.ed.gov/funding/21rfas.asp. Also, download the application package (search for CFDA 84.305) from https://grants.gov/. Contact Grants.gov (1-800-518-4726; support@grants.gov) if you need help with your electronic grant submission.

 

Next, take advantage of our digital technical assistance options.

  • On-demand webinars. These pre-recorded webinars answer questions about the grant competitions, how to apply, and how to prepare a strong application. You can access them here: https://ies.ed.gov/funding/webinars/.  

 

  • Virtual office hours. This year, we will host a series of drop-in hours during which a program officer will answer questions and give technical assistance. These office hours will help determine which competition or project type is the best fit and also understand some of the requirements and recommendations in the RFAs. Please see the schedule below along with the call-in information. This information is also posted here.

 

  • Cost analysis/Cost-effectiveness analysis. Many RFAs require a cost analysis plan, and some also require a cost effectiveness plan.  Please refer to our list of resources for developing these plans: https://ies.ed.gov/seer/cost_analysis.asp.

 

 

Finally, please make sure that you attend to the application due dates: https://ies.ed.gov/funding/futureComp.asp because IES does not accept late applications.

 

Virtual Office Hours

Staff from the research centers will host hour-long drop-in virtual sessions to provide technical assistance around particular competitions or research project types or for general purposes. Applicants are encouraged to join in the discussion and ask questions. These sessions are especially helpful if you are unsure of which competition or project type is the best match for you or if you are unclear on any changes to the requirements or recommendations. Below is a list of the current sessions and their topics. Please attend as many sessions as you would like.

All office hours will use the same call-in details. The program officer will allow participants into the meeting from the “lobby” at the beginning. We recommend you do not use video so that there is sufficient bandwidth. All times are shown in Eastern Standard time.

 

Join Microsoft Teams Meeting

+1 202-991-0393   United States, Washington DC (Toll)

Conference ID: 915 412 787#

 

If you would like to request accommodations (e.g., TTY), please send an email to NCER.Commissioner@ed.gov with this request as soon as possible.

You may have to download a free mobile application to use Microsoft Teams if you want the full audio and visual experience from your phone. Clicking on the linked “Join” hyperlink below should prompt you to do this. You can also refer to this article for information: https://support.microsoft.com/en-gb/office/set-up-your-teams-mobile-apps-1ba8dce3-1122-47f4-8db6-00a4f93117e8

 

 

Virtual Office Hours Schedule

 

 

Monday, June 22

Tuesday, June 23

Wednesday, June 24

Thursday, June 25

12:30 – 1:30 pm ET

Competition fit: this will cover all NCER grant competitions and items such as applicant eligibility, general requirements, submission questions, and the IES review process.

Efficacy/Follow-Up and Replication: this will cover characteristics of high-quality projects of these types.

Exploration projects: this will cover characteristics of high-quality projects of this type.

Development projects: this will cover characteristics of high-quality projects of this type.

2:00 – 3:00 pm ET

Exploration projects: this will cover characteristics of high-quality projects of this type.

Development projects: this will cover characteristics of high-quality projects of this type.

Is 305A (Education Research Grants) right for me? This will address general questions about CFDA 84.305A

Measurement projects: this will cover characteristics of high-quality projects of this type.

 

 

Monday, June 29

Tuesday, June 30

Wednesday, July 1

Thursday, July 2

12:30 – 1:30 pm ET

Development projects: this will cover characteristics of high-quality projects of this type.

Exploration projects: this will cover characteristics of high-quality projects of this type.

Measurement projects: this will cover characteristics of high-quality projects of this type.

 

2:00 – 3:00 pm ET

Competition fit: this will cover all NCER grant competitions and items such as applicant eligibility, general requirements, submission questions, and the IES review process.

Systematic Replication: this will focus on the requirements for a 305R or 324R application

Efficacy/Follow-Up: this will cover characteristics of high-quality projects of this type.

Pathways to the Education Sciences: this will address common questions about this training program opportunity.