Inside IES Research

The National Assessment of Educational Progress (NAEP) is the largest nationally representative and ongoing assessment of subject knowledge among students in public and private schools in the United States. On the 2017 eighth grade mathematics assessment, 38% of students without disabilities scored at the NAEP Proficient level or above while 25% scored below the NAEP Basic level. However, for students with disabilities, math achievement levels were much worse. Only about 9% of students with disabilities scored at the NAEP Proficient level or above whereas 69% scored below the NAEP Basic level. In response to this gap, in 2021, the National Center for Special Education Research (NCSER) released a funding opportunity to coincide with the release of the 2017 Grade 8 NAEP Mathematics response process data. NCSER intended to support research that explores how learners with disabilities interact with the NAEP digital assessment to better support these learners in test-taking environments and determine whether and how that information could be used to inform instructional practices. There is much to learn from research on NAEP process data for understanding test-taking behaviors and achievement of learners with disabilities. Below we showcase the latest findings from currently funded research and encourage more investigators to conduct research with newly released process data.

Since 2017, administrations of NAEP have captured a variety of response process data, including keystrokes as learners progress through the assessment, how learners use the available tools (such as the calculator), and how accommodations (for example, text-to-speech or more time to complete the assessment) affect performance. Besides score data, NAEP datasets also include survey data from learners, teachers, and schools, and information on test item characteristics and student demographics (including disability). Together, these data provide a unique opportunity for researchers to conduct an in-depth investigation of the test-taking behavior and the mathematics competencies of learners with disabilities compared to their peers without disabilities.  

In July 2021, IES awarded two grants to conduct research using NAEP process data. The results of these projects are expected to improve the future development and administration of digital learning assessments, identify needed enhancements to mathematics instruction, and highlight areas where further research is needed.  Although these projects are ongoing, we would like to highlight findings from one of the funded projects awarded to SRI International and led by principal investigator Xin Wei  entitled Analysis of NAEP Mathematics Process, Outcome, and Survey Data to Understand Test-Taking Behavior and Mathematics Performance of Learners with Disabilities.

The findings from this study, recently published in Autism, is an example of the power of process data to shed new light on learners with disabilities. Focusing on autistic students, Xin Wei and her team analyzed data from 15 items on the NAEP math assessment, their response time in seconds, their score on the items (including partially correct scoring), and survey data related to their enjoyment, interest, and persistence in math. They also analyzed the content of each item using Flesch Reading Ease scores to measure the reading difficulty level of the item. Finally, they rated each item based on the complexity of any social context of the item, as prior research has shown that these contexts can be more challenging for autistic students. They conducted statistical analyses to compare the performance of autistic students with extended time accommodations, autistic students without accommodations, and general education peers. The researchers were not only looking for any areas of weakness, but also areas of strength. Previous studies have demonstrated that autistic people frequently excel in abstract spatial reasoning and calculation tasks, relying more on visual-mental representations than verbal ones.

The findings showed that in comparison to their general education peers, unaccommodated autistic students scored higher and solved math problems involving the identification of figures more quickly. Unaccommodated autistic students were also faster than their general education peers at solving the following types of math items: comparing measures using unit conversions, mentally rotating a triangle, interpreting linear equations, and constructing data analysis plots. Although autistic students who used the extended-time accommodation were lower performing than the other two groups, they had a higher accuracy rate on items involving identifying figures and calculating the diameter of a circle. Both groups of autistic students seem to perform poorer on word problems. Researchers concluded that the linguistic complexity could be one of the reasons that autistic students struggle with math word problems; however, there were two word problems with which they seemed to struggle despite the fact that they were not linguistically complex. It turns out that the items were rated as having substantial social context complexity. The researchers also looked at the student survey data on what types of math they enjoyed more and found they had more enjoyment working with shapes and figures and less enjoyment for solving equations.

The researchers recommend incorporating meta-cognitive and explicit schema instruction during mathematics instruction to aid autistic students in understanding real-life math word problems. They also recommend that assessment developers consider simplifying the language and social context of math word problems to make the assessment more equitable, fair, and accessible for autistic students. Because the autistic student population is particularly heterogenous, more research is required to better understand how to improve instructional strategies for them.

IES plans to release the same type of process data from the 2017 Grade 4 NAEP Mathematics at the end of this summer. We encourage researchers to request these process data to conduct research to understand test-taking behavior and performance of students with disabilities at the elementary school level. For a source of funding for the work, consider applying to the current Special Education Research Grants competition. Here are some important resources to support your proposal writing:

• Conducting research using process data from NAEP
• IES/NCSER Standard Research Grant Program (applications due September 21, 2023)
• Response Process Data From the 2017 NAEP Grade 4 and 8 Mathematics Assessment
• Apply for NAEP restricted-use datasets

This blog was authored by Sarah Brasiel (Sarah.Brasiel@ed.gov), program officer at NCSER, and Juliette Gudknecht, summer data science intern at IES and graduate student at Teachers College, Columbia University. IES encourages special education researchers to use NAEP response process data for research under the Exploration project type within our standard Special Education Research Grants Program funding opportunity.   

On May 16-18, 2023, NCER and NCSER hosted our second virtual Principal Investigators (PI) Meeting. Our theme this year was Building on 20 Years of IES Research to Accelerate the Education Sciences. Because it was the IES 20^th anniversary this past year, we used this meeting as an opportunity to reflect on and celebrate the success of IES and the education research community. Another goal was to explore how IES can further advance the education sciences and improve education outcomes for all learners.

Roddy Theobald (American Institutes for Research) and Eunsoo Cho (Michigan State University) graciously agreed to be our co-chairs this year. They provided guidance on the meeting theme and session strands and also facilitated our plenary sessions on Improving Data on Teachers and Staffing Challenges to Inform the Next 20 Years of Teacher Workforce Policy and Research and the Disproportionate Impact of COVID-19 on Student Learning and Contributions of Education Sciences to Pandemic Recovery Efforts. We want to thank them for their incredible efforts in making this year’s meeting a big success!

Here are a few highlights:

The meeting kicked off with opening remarks from IES Director, Mark Schneider, and a welcome from the Secretary of Education, Miguel Cardona. Director Schneider spoke about the importance of timeliness of research and translation of evidence to practice. IES is thinking about how best to support innovative approaches to education research that are transformative, embrace failure, are quick turnaround, and have an applied focus. He also discussed the need for data to move the field forward, specifically big data researchers can use to address important policy questions and improve interventions and education outcomes. Secretary Cardona acknowledged the robust and useful evidence base that IES-funded researchers have generated over the last 20 years and emphasized the need for continued research to address historic inequities and accelerate pandemic recovery for students.

This year’s meeting fostered connections and facilitated deep conversations around meaningful and relevant topic areas. Across the three day PI Meeting, we had over 1,000 attendees engaged in virtual room discussions around four main topic areas (see the agenda for a complete list of this year’s sessions):

• Diversity, Equity, Inclusion, and Accessibility (DEIA)—Sessions addressed DEIA in education research
• Recovering and Learning from the COVID-19 Pandemic—Sessions discussed accelerating pandemic recovery for students and educators, lessons learned from the pandemic, and opportunities to implement overdue changes to improve education
• Innovative Approaches to Education Research—Sessions focused on innovative, forward-looking research ideas, approaches, and methods to improve education research in both the short- and long-term
• Making Connections Across Disciplines and Communities—Sessions highlighted connections between research and practice communities and between researchers and projects across different disciplines and methodologies

We also had several sessions focused on providing information and opportunities to engage with IES leadership, including NCER Commissioner’s Welcome; NCSER Acting Commissioner’s Welcome; Open Science and IES, NCEE at 20: Past Successes and Future Directions; and The IES Scientific Review Process: Overview, Common Myths, and Feedback.

Many  sessions also had a strong focus on increasing the practical impacts of education research by getting research into the hands of practitioners and policymakers. For example, the session on Beyond Academia: Navigating the Broader Research-Practice Pipeline highlighted the unique challenges of navigating the pipeline of information that flows between researchers and practitioners and identified strategies that researchers could implement in designing, producing, and publishing research-based products that are relevant to a broad audience. The LEARNing to Scale: A Networked Initiative to Prepare Evidence-Based Practices & Products for Scaling and The Road to Scale Up: From Idea to Intervention sessions centered around challenges and strategies for scaling education innovations from basic research ideas to applied and effective interventions. Finally, the Transforming Knowledge into Action: An Interactive Discussion focused on identifying and capturing ways to strengthen dissemination plans and increase the uptake of evidence-based resources and practices.  

We ended the three-day meeting with trivia and a celebration. Who was the first Commissioner of NCSER? Which program officer started the same day the office closed because of the pandemic? Which program officer has dreams of opening a bakery? If you want to know the answers to these questions and more, we encourage you to look at the Concluding Remarks.  

Finally, although we weren’t in person this year, we learned from last year’s meeting that a real benefit of having a virtual PI meeting is our ability to record all the sessions and share them with the public. A part of IES’s mission is to widely disseminate IES-supported research. We encourage you to watch the recorded sessions and would be grateful if you shared it with your networks.

We want to thank the attendees who made this meeting so meaningful and engaging. This meeting would not have been a success without your contributions. We hope to see our grantees at the next PI Meeting, this time in-person!

If you have any comments, questions, or suggestions for how we can further advance the education sciences and improve education outcomes for all learners, please do not hesitate to contact NCER Commissioner Liz Albro (Elizabeth.Albro@ed.gov) or NCSER Acting Commissioner Jackie Buckley (Jacquelyn.Buckley@ed.gov). We look forward to hearing from you.

The IES director recently posted a blog indicating that IES had to return approximately $44 million in unobligated funds of the $100 million total American Rescue Plan (ARP) funding IES received to help the nation's students recover from the learning losses of the pandemic. NCSER was hard hit by the rescission of these funds.

As transparent as we try to be, admittedly, the federal budgeting process is not always clear. Many of you have reached out with concerns about the potential impact of these ARP rescissions on your current grants and future funding opportunities. Please allow me to explain the current context of fiscal year 2023 funding and forecast for fiscal year 2024.

NCSER's Grant Funding: Where the Money Comes From and How It Is Spent

NCSER funds come from the Research in Special Education (RiSE) appropriation, which is one small part of the larger IES appropriations account. RiSE supports all of NCSER’s typical grant competitions. We also contribute money from this account to our share of other IES activities such as the grant peer review process and the PI meeting.

As those of you who have been funded by NSCER know, we provide grant funding on an annual basis. Even though we fund projects annually, once we make an award, we are committed to providing annual costs for a continuing project through the duration of the designated study period. Consequently, the amount of money available to support new research and training awards each year is contingent, in part, upon the number of current awards and their outyear costs. Any time NCSER funds a high number of new awards (and thereby commits to funding every award through the duration of the designated study period), there will be less money available for new awards the following year, unless the RiSE program appropriation receives an increase from Congress.

Deciding what new grant competitions in NCSER might look like in any given year requires that we balance many factors, including: (1) the amount of funding Congress is likely to appropriate to RiSE (note that we typically have to make decisions before we know for sure how much money we will have), (2) projected continuation costs for existing awards and commitments, (3) estimates of total funding available for new awards based on 1 and 2 above, and (4) a best guess prediction of the percent of applicants that will be successful, based on trends over time, in any single NCSER sponsored competition. If you have been around NCSER long enough, you know our funding is typically very tight, sometimes so tight we can’t offer any competitions (FY 2014) or need to significantly limit available competitions (FY 2017).

NCSER’s ARP-funded Research Projects

In FY 2022, once again, we found ourselves with insufficient funds to hold our typical special education research grant competitions. At the same time, IES received $100 million in ARP funding. With NCSER’s share of those funds, we chose, in part, to hold a new Research to Accelerate Pandemic Recovery in Special Education grants program to fund projects that addressed the urgent challenges faced by districts and schools in supporting learners with or at risk for disabilities, their teachers, and their families in the aftermath of the pandemic. The competition was funded solely using ARP funds. To be clear, RiSE funds were never intended to be a source of funding for these projects. 

By now you may be predicting where this blog is going…

NCSER was thrilled to be able to fund 9 research grants through this ARP-funded competition, all of which have the potential to improve outcomes significantly and rapidly for students with or at risk for disabilities. A little less than 2 months ago, NCSER was in the process of documenting annual progress and approving continuation funding for these grantees when the ARP funds were unexpectedly rescinded (returned to the U.S. Treasury as part of the debt ceiling deal). These projects were in various stages of progress, but each was just finishing the first year of the grant and it is fair to say that, overall, a significant amount of work (and grant costs) remained at the time of this rescission.

As I mentioned, NCSER has operated from the perspective that when we make a commitment to funding your grant, we prioritize your continuation costs first before funding new awards or initiatives. In other words, if we are ever in a budget crunch, we will meet our existing commitments first before using money on new activities. Although the ARP funding source was eliminated, our commitment to those FY 2022 ARP-funded grants remained. We chose to use money from our RiSE account to pay for current and future continuation costs for these grants. I hope everyone can understand that this difficult decision honors our standard practice of prioritizing existing commitments.

NCSER’s FY 2023 Research Competition

After accounting for the cost of the continuations that would have otherwise been supported using ARP funds, NCSER’s ability to fund new awards in the FY 2023 grant competition was limited. Further exasperating our new budget shortfall was the much higher than expected (based on past application and funding trends) number of FY 2023 applications that were rated outstanding or excellent. This is a great testament to the field and the work that you all do! Unfortunately, this success came at the same time as this unexpected, very large budget rescission. Something had to give and sadly, what gave was our ability to fund many worthy new grants. It was not a decision made lightly or without thought for those grants left unfunded. I know that many of you are disappointed in this outcome.

It takes a tremendous amount of effort to produce a grant application and we recognize your continued efforts to work with NCSER staff throughout the pre-award process. It is heartbreaking to find out a grant you submitted won’t be funded, despite having such a strong score. NCSER staff were heartbroken with you.

Outlook for FY 2024

What does this all mean for NCSER moving forward? Despite the setback this year, based on available information we have now, NCSER plans to offer research competitions in FY 2024. We are committed to offering new funding opportunities whenever possible to continue the tremendous strides we have made in improving the depth, breadth, and quality of special education research in this country.

NCSER and NCER will be notifying the field very soon regarding FY 2024 competitions, so stay tuned. If you have not done so already, please sign up for our Newsflash to stay current on IES happenings, including the release of new funding opportunities.

Although the challenges we experienced this year certainly were disappointing, I want to end on what I see as the silver lining that emerged from of all of this. Namely, since NCSER’s first research competitions in 2006, the capacity in the field to conduct high-quality research and carry out excellent research training has grown tremendously. We should not forget how far we have come, and how bright NCSER’s future is. Our funding has not (yet!) kept pace with that growth, but that is a subject for another blog…

Please reach out to me at Jacquelyn.Buckley@ed.gov with questions or comments. I'm always happy to hear from you!

One year ago, IES introduced a new equity standard and associated recommendations to its Standards for Excellence in Education Research (SEER). The intent of this standard, as well as the other eight SEER standards, is to complement IES’s focus on rigorous evidence building with guidance and supports for practices that have the potential to make research transformational. The addition of equity to SEER is part of IES’s ongoing mission to improve academic achievement and access to educational opportunities for all learners (see IES Diversity Statement). IES is mindful, however, that to authentically and rigorously integrate equity into research, education researchers may need additional resources and tools. To that end, IES hosted a Technical Working Group (TWG) meeting of experts to gather input for IES’s consideration regarding the existing tools and resources that the education community could use as they implement the new SEER equity standard in their research, along with identifying any notable gaps where tools and resources are needed. A summary of the TWG panel discussion and recommendations is now available.

The TWG panel recommended several relevant resources and provided concrete suggestions for ways IES can support education researchers’ learning and growth, including training centers, coaching sessions, webinars, checklists, and new resource development, acknowledging that different researchers may need different kinds of supports. The meeting summary includes both a mix of recommendations for tools and resources, along with important considerations for researchers, including recommendations for best practices, as they try to embed equity in their research. 

The new SEER equity standard and accompanying recommendations have been integrated throughout the current FY 2024 Request for Applications. By underscoring the importance of equity, the research IES supports will both be rigorous and relevant to address the needs of all learners.   

This blog was written by NCER program officer Christina Chhin. If you have questions or feedback regarding the equity TWG, please contact Christina Chhin (Christina.Chhin@ed.gov) or Katina Stapleton (Katina.Stapleton@ed.gov), co-chair of the IES Diversity Council. If you have any questions or feedback regarding the equity standard or associated recommendations, please email NCEE.Feedback@ed.gov.

On April 25, 2022, the IES research centers held a listening session focused on researchers with disabilities. The purpose of the session was for participants to share their experiences in becoming education researchers and applying for/carrying out research grants as well as to offer suggestions for increasing the participation of individuals with disabilities in IES grant programs. Participants included researchers from institutes of higher education and non-profit research agencies, researchers in training, higher education administrators and staff, and staff from the Department of Education and other federal funding agencies. The discussion centered around three questions. Below is a summary of the key themes that participants highlighted in response to each question.

How has your disability, in conjunction with other intersecting identities, shaped your experiences as a researcher?

Disability experiences can shape research careers. Participants described an evolving sense of identity and how that impacts their research trajectories. For example, one participant described how conducting disability research helped them recognize their own experiences with mental illness, attention-deficit hyperactivity disorder, and autism: “That shifting understanding of my own identities […] has been something that informs my own thinking about the research and my own paths.”

Ableism affects many researchers with disabilities. Several participants mentioned experiencing ableism, a form of discrimination against people with disabilities based on the belief that typical abilities are superior. As one participant with a physical disability explained, “There's a variation in productivity, and it's been very hard finding a community of other academics because support and advice for graduate students, for example, all presume able body.” Participants also noted that ableism is particularly salient for academics with disabilities who are marginalized in other ways as well (for instance, based on their race or ethnicity, gender identity, and/or sexual orientation). 

Accommodations for researchers with disabilities are inadequate. Participants noted, “I’ve had multiple situations where the accessibility office just does not know how to handle disabled graduate students outside of classes” and “these same issues can follow researchers to the faculty stage of their careers, only then they must work with HR [human resources], yet another entity who is unfamiliar with how to accommodate faculty disability-related requests.” Another participant emphasized how this applies to people whose disabilities are not visible, explaining that individuals may attempt to hide their disabilities, but then they may not find out about accommodations that would have been available to them; on the other hand, “the accommodations they receive don't really help them to be productive and attain their scholarly research expectations that they have.”

Participants feel a responsibility toward people with disabilities in their research. For example, one participant shared that, “As a deaf woman of color, I feel the responsibility to conduct research that elevates and addresses significant issues of need in the community. I also feel the need to protect the community from hearing researchers that conduct research based on what they determine to be their definition of the quality of life.”

How has your disability, in conjunction with other intersecting identities, impacted your experiences applying for and conducting an IES research grant?

Having a disability can be an asset to research. As one participant described, “I do believe that having this learning disability myself has impacted the way that I conceptualize mathematical thinking and understand the ways that other people might conceptualize mathematical thinking.” Participants also discussed the importance of involving researchers with disabilities in research focused on individuals with disabilities. As one participant stated, “I feel that it's very important to bring that insider's view to people about the process of learning how to read, and that's one big gap in literacy [research] – a lot of people have not done the research on deaf children and their development, and the people doing that are not deaf themselves, so they don't have that firsthand experience, that understanding.”  

IES grant timelines are not always suited for researchers with disabilities. As one participant noted, “Disability is fluid, it's not always the same… and there can be difficulty in predicting certain things, and even in just figuring out what kind of things I needed to be able to do the research.” Another participant added, “Being able to get grants, support, etc. is all presumed upon working on a non-sick person's timeline and standard of productivity.”

The IES peer review process may present a barrier to certain types of research on learners with disabilities. For example, one deaf participant was concerned about peer reviewers being able to review their proposal focused on American Sign Language with impartiality, given as they noted, “a strong audio-centric bias within the field.” Another participant shared that, “Most of my attempts to submit applications have favored individuals who conduct RCTs [randomized controlled trials] or other more quantitative focused research. As a researcher with a disability who believes the voices need to be heard/represented, I find the IES focus for grants to be limiting.”

Requests for applications (RFAs) and federal register notices should be more accessible. For instance, RFAs in PDF format can be difficult for people with visual impairments to take notes in and navigate with screen readers. Because of varying needs and preferences, participants recommended making application-related documents available in multiple formats.

How can IES build the research capacity of students, researchers, and organizations from various disability communities?

Participants emphasized a need for more researchers with disabilities to receive grants, which in turn would provide more opportunities for students with disabilities to be involved in research. According to one participant, “I have deaf students, and I would like to pull them into the field as well and have them become experienced researchers.” Increasing capacity in the field would also involve people with disabilities serving on peer review panels. As one participant noted, “People with disabilities absolutely need to be part of that [review] process, and definitely need to be tied to the disability group that the content is for.”

Participants suggested IES develop training and mentorship programs for researchers with disabilities. IES could also consider providing diversity supplements like the National Institutes of Health to fund postdoctoral positions on active grants. Other suggestions for IES included checking for biased assumptions in RFAs and ensuring the language empowers researchers who experience disabilities.

IES has taken steps to respond this feedback, including:

• Publishing RFAs in multiple formats
• Including recommendations in the RFA for researchers to incorporate input from learners, educators, and other stakeholders at every stage of the research
• Requiring that training grants budget for accommodations for trainees with disabilities
• Featuring researchers with disabilities in the IES blog
• Awarding a contract to identify best practices for broadening participation in grant programs

This blog was authored by Katherine Taylor (Katherine.Taylor@ed.gov), NCSER program officer, with feedback from IES program officers Katina Stapleton (Katina.Stapleton@ed.gov) and Akilah Nelson (Akilah.Nelson@ed.gov). Thematic coding of the listening session transcript was completed by IES interns, Kaitlynn Fraze and Alysa Conway, with support from Katherine Taylor.