IES Blog

Institute of Education Sciences

Developing an Evidence Base for Researcher-Practitioner Partnerships

I recently attended the annual meeting of the National Network of Education Research-Practice Partnerships. I was joined by well over 100 others who represented a wide swath of partnerships (RPPs), most supported by IES funds.  When it comes to research, academic researchers and practitioners often have different needs and different time frames. On paper, RPPs look like a way to bridge that divide.

Over the last few years, IES has made some large investments in RPPs. The Institute’s National Center for Education Research runs an RPP grant competition that has funded over 50 RPPs, with an investment of around $20 million over the last several years. In addition, the evaluation of state and local programs and policies competition has supported partnerships between researchers and state and local education agencies since 2009.

But the biggest investment in RPPs, by far, has been through the Regional Educational Laboratories. In the 2012-2017 REL funding cycle, 85 percent of the REL’s work had to go through “alliances”, which often coordinated several RPPs and themselves emphasized research to practice partnerships. In the current funding cycle, RELs have created over 100 RPPs, and the bulk of REL’s work—upwards of 80 percent—is done through them.

Back of the envelope calculations show that IES is currently spending over $40 million per year on REL RPPs. Add to that the hundreds of millions of dollars invested in alliances in the previous REL contract plus the RPP and state policy grant competitions and this constitutes a very big bet.

Despite the fact that we have invested so much in RPPs for over half a decade, we have only limited evidence about what they are accomplishing.

Consider the report that was just released from the National Center for Research in Policy and Practice. Entitled A Descriptive Study of the IES Researcher-Practitioner Partnership Program, it is exactly what it says it is: a descriptive study.  Its first research goal focused on the perceived benefits of partnerships and the second focused on partnership contexts.

But neither of these research goals answers the most important question: what did the partnerships change, not just in terms of research use or service delivery, but in what matters the most, which is improved outcomes for students.

Despite IES’ emphasis on evidence-based policy, right now RPPs are mostly hope-based. As noted, some research has documented a few of the processes that seem to be associated with better functioning RPPs, such as building trust among partners and having consultative meetings. Research has not, however, helped identify the functions, structures, or processes that work best for increasing the impact of RPPs.

The Institute is planning an evaluation of REL-based RPPs. We know that it will be difficult and imperfect. With over $200 million invested in the last REL cycle, with over 100 REL-based RPPs currently operating, and with $40+ million a year supporting RPPs, we assume that there’s lots of variation in how they are structured, what they are doing, and ultimately how successful they are in improving student outcomes. With so many RPPs and so much variation, our evaluation will focus on the “what works for whom and under what circumstances” type questions: Are certain types of RPPs better at addressing particular types of problems? Are there certain conditions under which RPPs are more likely to be successful?  Are there specific strategies that make some RPPs more successful than others?  Are any successful RPP results replicable?

Defining success will not be simple. A recent study by Henrick et al. identifies five dimensions by which to evaluate RPPs—all of which have multiple indicators. Since it’s not likely that we can adequately assess all five of these dimensions, plus any others that our own background research uncovers, we need to make tough choices. Even by focusing on student outcomes, which we will, we are still left with many problems. For example, different RPPs are focused on different topics—how can we map reasonable outcome measures across those different areas, many of which could have different time horizons for improvement?

Related to the question of time horizons for improvement is the question of how long it takes for RPPs to gain traction. Consider three of arguably the most successful RPPs in the nation: The Chicago Consortium was launched in 1990; the Baltimore consortium, BERC, in fall 2006; and the Research Alliance for New York City Schools in 2008. In contrast, IES’ big investment in RPPs began in 2012. How much time do RPPs need to change facts on the ground? Since much of the work of the earliest alliances was focused on high school graduation rates and college access, 6 years seems to be a reasonable window for assessing those outcomes, but other alliances were engaged in work that may have longer time frames.

The challenges go on and on. But one thing is clear: we can’t continue to bet tens of millions of dollars each year on RPPs without a better sense of what they are doing, what they are accomplishing, and what factors are associated with their success.

The Institute will soon be issuing a request for comments to solicit ideas from the community on the issues and indicators of success that could help us inform our evaluation of the RPPs. We look forward to working with you to provide a stronger evidence base identifying what works for whom in RPPs.

Mark Schneider
Director, IES

Informing Future Research in Career and Technical Education (CTE)

Career and Technical Education (CTE) has been evolving and expanding at a rapid pace in recent years as industry and education leaders focus on students’ readiness for college and careers. While some studies have shown positive effects of CTE on students, the evidence base is thin. To learn more about the research needs of the CTE field, the National Center for Special Education Research (NCSER) and the National Center for Education Research (NCER) at the Institute of Education Sciences (IES) convened a group of experts in policy, practice and research related to CTE.  The discussion held by the Technical Working Group (TWG) led both NCER and NCSER to increase their investments in CTE for fiscal year 2019: NCSER included a CTE special topic, and NCER changed its CTE topic from a special topic to a standing topic. Applications to both are due August 23, 2018.  Both research centers hope to fund more studies that will help us better understand this growing aspect of education.

The TWG focused on the following four questions:

  1. Who is served by CTE and who is left behind? From national CTE statistics, we know that 82% of all public high schools offer CTE. And, 85% of students earn at least one credit in CTE with the average high school student earning 2.5 CTE credits. However, TWG members noted that research is lacking on specific subpopulations in CTE, such as students from various demographic backgrounds and students with disabilities. Disaggregated data on these dimensions are needed to better understand the CTE experiences of the range of students being served. Such data may help educators improve equity of access to high quality programs for all students.
  2. What do we know―and need to know―about CTE policies, programs, and practices at the secondary and postsecondary levels?  TWG experts discussed the need to know more about industry-recognized credentials and about business and industry engagement in CTE at the secondary level. They argued that we do not know if credentials align with industry requirements, nor do we understand the impact of different types of credentials on student outcomes and wage trajectories. TWG members also noted that the higher the perceived quality or prestige of the CTE program, the more exclusive it becomes, and the more difficult it is for disadvantaged students to obtain access. TWG members also expressed concerns about CTE teacher training, particularly for experts who are recruited from industry without prior teacher preparation. As the experts discussed postsecondary CTE, they suggested that the field would be best served by framing the conversation about secondary to postsecondary pathways as a continuum that enables transparent and sequential transitions from secondary to 2-year and then to 4-year programs or to training or employment, with guidance for students to understand possible sequences.
  3. What are the critical methodological issues in CTE?  TWG members noted that, with a few notable exceptions (e.g.,  a 2008  MDRC study on career academies in New York and a recent study of CTE high schools in Massachusetts), few causal studies on CTE have been conducted. There is an urgent need for more high quality, causal research on CTE policies and programs. In addition, the experts noted that there is almost no research on students with disabilities in CTE. TWG members concluded that the field needs to re-conceptualize CTE research – including better defining CTE students, instructors, programs, and measures – and identify the critical research questions in order to encourage more research in this field.
  4. What is needed to advance CTE research?  State CTE administrators want to know how to identify quality CTE programs so they know how to spend their dollars most effectively on programs that best meet the needs of students. Policymakers also want to know what “works” and what the benefits are of such investments. The TWG members encouraged studies that examine the educational benefits of particular instructional approaches. They also highlighted the importance of collaborative cross-institutional and cross-agency efforts to advance CTE research.

Readers are invited to read the summary of the TWG discussion.

By Corinne Alfeld (NCER program officer) and Kimberley Sprague (former NCSER program officer)

Connect with NCES Researchers at Upcoming Summer Conferences: STATS-DC, JSM and ASA

NCES staff will share their knowledge and expertise through research presentations, training sessions, and booth demonstrations at three notable conferences this summer (listed below). The NCES booth will be also be featured at exhibit halls where conference attendees can “ask an NCES expert,” learn how NCES data can support their research, or pick up publications and products.

NCES STATS-DC Data Conference

July 25 – 27
Washington, DC
The Mayflower Hotel

STAT-DC is NCES’ annual conference designed to provide the latest information, resources and training on accessing and using federal education data. Researchers, policymakers and data system managers from all levels are invited to discover innovations in the design and implementation of data collections and information systems. There is no registration fee to attend STATS-DCparticipants must complete registration paperwork onsite at the conference.

Key conference items:

  • Learn updates on federal and state activities affecting data collection and reporting, with a focus on the best new approaches in education statistics
  • Attend general information sessions on CCD, data management, data use, and data privacy, etc.
  • Partake in trainings for Common Core of Data (CCD) and EDFacts data coordinators
  • Attend data tools and resource demonstrations from NCES staff during designated times at the NCES exhibit booth.

NCES Staff Presentations:

Explore the full conference agenda. Some highlighted sessions are shown below.

WEDNESDAY, JULY 25

9:00 a.m. – 12:00 noon
Common Core of Data (CCD) Fiscal Coordinators' Training
District Ballroom

1:15 p.m. – 2:15 p.m.
Opening Plenary Session by Dr. Lynn Woodworth
Grand Ballroom

4:30 p.m. – 5:20 p.m.
Introduction to the Common Core of Data: America's Public Schools by Mark Glander
Palm Court Ballroom

9:00 a.m. – 12:30 p.m.
EDFacts and Common Core of Data (CCD) Nonfiscal Coordinators’ Training
Grand Ballroom

THURSDAY, JULY 26

9:00 a.m. – 10:00 a.m.
Title I Allocations by Bill Sonnenberg
Palm Court Ballroom

 

American Statistical Association – Joint Statistical Meetings

July 28 – August 2
Vancouver, British Columbia, Canada
Vancouver Convention Centre

JSM is the largest gathering of statisticians and data scientists in North America. Exchange ideas and explore opportunities for collaboration across industries with NCES staff and other statisticians in academia, business, and government.
 

Key conference items:

  • Review applications and methodology of statistics, such as analytics and data science
  • Attend technical sessions, poster presentations, roundtable discussions, professional development courses and workshops
  • Visit the NCES booth in the exhibit hall booth #227 and meet the NCES Chief Statistician, Marilyn Seastrom.

NCES Staff Presentation:

THURSDAY, AUGUST 2
8:35 a.m. – 10:30 a.m.
Educating the Government Workforce to Lead with Statistics by Andrew White
CC-East 10

 

American Sociological Association – Annual Meeting

August 11 – August 14
Philadelphia, Pennsylvania
Pennsylvania Convention Center and the Philadelphia Marriott Downtown

Professionals involved in the scientific study of society will share knowledge and discuss new directions in research and practice during this annual meeting.

Key conference items:

  • Choose from 600 program sessions throughout the 4-day conference
  • Browse and discuss topics from 3,000+ research papers submitted
  • Swing by the NCES exhibit hall booth #211

 

Follow us on twitter (@EdNCES) throughout these upcoming conferences to stay up to date and learn the latest in education statistics. We hope you’ll join us whether in person or online!

ED/IES SBIR Awardee Leads Event Featuring Live Conversation with a NASA Astronaut in Space

On Wednesday June 27, 2018, the Smithsonian’s National Air and Space Museum held Space Innovation Day, an event to celebrate space exploration, STEM education, and students as makers. The event was co-developed by the museum and Future Engineers, a technology firm that is a current awardee of the U.S. Department of Education and Institute of Education Sciences’ Small Business Innovation Research Program (ED/IES SBIR). 

In the morning, the event featured a live conversation (called a “downlink”) between NASA astronaut Serena Auñón-Chancellor on the International Space Station and Washington, D.C.-area students at the museum. After a brief introduction of Auñón-Chancellor as she floated around in the space station, students asked her a series of questions such as “What it is like to experience space?” and “What does it take to be an astronaut?”

The morning also included on-stage interviews with three students who won the Future Engineers Two For the Crew ChallengeThrough this national competition, sponsored by the ASME Foundation with technical assistance from NASA, K-12 students submitted a digital design of an astronaut tool intended to be manufactured on the International Space Station using a 3-D Printer. This tool allows innovative solutions to be provided to the astronauts immediately and means that NASA does not need to ship tools into space. One of the student winners designed “2 Pliers + 1 Handle,” a set of tool parts including needle-nose and lineman’s pliers with attachable handles. The 3-D printed multi-purpose tool can be customized into many different configurations when in space.

The challenge competition was run through a web-based platform that Future Engineers is developing with the support of a 2017 award from ED/IES SBIR.  The platform provides an online hub for students to create and submit solutions to innovation design challenges. Future Engineers is planning to launch the school version of their platform in the 2018-19 school year, with the goal of bringing many different kinds of maker design challenges to classrooms around the country across many areas of STEM for grades K to 12.

The afternoon of the event featured hands-on exhibits with educational opportunities for hundreds of students and museum attendees, including a 3-D design makerspace by Future Engineers, an augmented reality solar system experience by the Space Foundation, and a virtual reality space station experience by NASA.

We look forward to more maker design challenge events in the future!

Edward Metz is a program officer at the Institute of Education Sciences.

 

About ED/IES SBIR

The U.S. Department of Education’s Small Business Innovation Research program, administered by the Institute of Education Sciences (IES), funds projects to develop education technology products designed to support students, teachers, or administrators in general or special education. The program emphasizes rigorous and relevant research to inform iterative development and to evaluate whether fully-developed products show promise for leading to the intended outcomes. The program also focuses on commercialization once the award period ends so that products can reach students and teachers and be sustained over time. ED/IES SBIR-supported products are currently used in thousands of schools around the country.

 

 

 

Results from a Study of the IES Researcher-Practitioner Partnership Grants

Funding research that makes it into classrooms and impacts the lives of students is an important goal for IES. One of the ways in which the Institute strives to make sure the work it funds is useful to education practitioners is through the funding of Researcher-Practitioner Partnerships (RPPs), which are grants to support researchers and education agencies working together to answer questions of high priority for the education agency. IES began funding these RPP projects in 2013 with the goal of supporting partnership development, initial research on problems of high priority for the education agency, and increasing the education agency’s capacity to use and understand research.

Because the RPP program is relatively new and little is known about the best way to frame and support these partnerships, IES funded the National Center for Research in Policy and Practice (NCRPP; an IES-funded national research and development center on knowledge utilization) to conduct a supplemental study to look at how researcher and practitioner partners form and maintain their partnerships, and strategies they use to increase research use by practitioners at the education agency. NCRPP also wanted to understand more about the IES RPP grant program in terms of funding amount, time frame, and Institute leadership and monitoring.

NCRPP just released its final report on the findings from the RPP study. In two waves of data collection, NCRPP distributed surveys and conducted interviews with key researchers and practitioners involved in 27 of the 28 RPPs funded by IES between 2013 and 2015. Findings from the report show that partnerships reported progress on goals related to developing findings that apply to other organizations and improving students’ socio-emotional outcomes. Additionally, researchers and practitioners both reported valuing their participation in the partnership work. Finally, when compared to a national sample of school and district leaders, practitioners who participate in an IES-funded RPP are much more likely to name journal articles as useful pieces of research and to indicate they integrate research processes into their own work.

You can read a summary about the findings on the NCRPP website here, or download the full interim report here.

Becky McGill-Wilkinson, NCER Program Officer for the Knowledge Utilization Research and Development Centers