Inside IES Research

Notes from NCER & NCSER

The Nexus Between Teaching and Research: What I Learned Working on an IES Grant

 

Samuel Choo is a doctoral student at the dissertation stage in the Department of Early Childhood, Special Education, and Rehabilitation Counseling at the University of Kentucky (UK). In this blog post, he describes how working on an IES grant gave him first-hand experiences in planning and carrying out research in schools. He also discusses how these research experiences helped him understand the important connections between research and teaching.

How did you get started working on this IES research project?

The first I heard of IES was six years ago as a resource room teacher at a middle school. Dr. Brian Bottge, who is now my doctoral adviser, was awarded a NCSER grant to test the effects of Enhanced Anchored Instruction (EAI) on the math performance of middle school students. My school was randomly assigned to the EAI group. The project staff did a good job of teaching us how to implement EAI in our resource rooms. Soon after teaching with the new curriculum, I noticed that my students were much more motivated and engaged than they had been. In fact, they looked like they were actually enjoying math! Posttest scores showed positive results in favor of the new curriculum.

And so this experience as a teacher got you more interested in research?

Yes! The next year I applied to the UK doctoral program. I joined Dr. Bottge’s IES grant team as a research assistant where I learned how classroom-based research is planned and conducted. I had many opportunities to participate in the research experience. In my case, I helped train math and special education teachers, observed classrooms and assessed research fidelity, provided teachers with technical support, assisted in scoring tests, and worked on data entry and analysis. Project leaders also asked me to suggest revisions to the daily lesson plans based on my experiences teaching with EAI the year before.

Can you talk more about your developing research interests related to math education?

After the grant ended and after I finished my doctoral coursework, I went back to teaching in North Carolina, where I taught low performing middle school students in a Title I resource room. I ran my own pilot studies using what I had learned while teaching with EAI as both a research participant and research assistant. To help offset the cost of materials for my first study, I was awarded a $1500 Bright Ideas Grant from the North Carolina’s Electric Cooperatives. Thanks to the company’s generosity, I was able to fully implement all the lesson plans developed by Dr. Bottge’s grant team.

This experience was especially important to me because it was my first try at conducting my own research with a prescribed protocol, which I had learned from working on the IES project. Posttests showed statistically significant improvement of students in the EAI group in both computation and problem solving. Based on these results, the sponsor invited me to participate in a panel discussion in Raleigh, NC. The CEOs of the company attended the event along with policy makers and school administrators from across the state. This whole process, from applying for funding to carrying out the study to reporting the results, helped me make connections between university, classroom, and community.

What have been your big takeaways from these experiences?

From the training I received as a study participant, I have become a better teacher.  From working on an IES-funded grant team, I learned a lot about how to conduct classroom-based studies. I am looking forward to designing new instructional methods and testing their effectiveness. Similar to how my students learned math in a hands-on way, I learned research methods by having the opportunity to use them in practice, and for that I am very grateful. 

Developing School-Wide Approaches for Bullying Prevention: The Value of Partnerships

By Katherine Taylor (NCSER Program Officer) and Emily Doolittle (NCER Program Officer)

About 22% of 12 to 18 year olds report being bullied at school. Bullying behavior can be obvious (pushing, name calling, destroying property) or more subtle (rumor spreading, purposeful excluding). In whatever form it takes, bullying involves acts of physical, verbal, or relational aggression that are repeated over time and involve a power imbalance. Bullying has a variety of harmful effects, including the potential for a negative impact on student academic achievement. This leads to the question, what can schools do to prevent bullying? In support of Bullying Prevention Awareness Month, we want to highlight two IES projects that have tackled this issue by developing programs that support social and problem-solving skills for students and a positive school climate.

In one project, Drs. Terri Sullivan and Kevin Sutherland at Virginia Commonwealth University developed and tested a school-wide violence prevention model for middle school students, with a special focus on youth with disabilities. The resulting model incorporates elements of a social-emotional skill-building program, Second Step, and a comprehensive bullying prevention program, the Olweus Bullying Prevention Program.

In a second project, Dr. Stephen Leff at the Children’s Hospital of Philadelphia developed and is currently testing the Partner for Prevention (P4P) program to address aggression and bullying in elementary schools. P4P includes a classroom program, consultation for teachers and playground and lunchroom staff, and community outreach to engage parents in efforts to address bullying.

In both projects, the initial development work was accomplished using a community-based participatory research framework. Both projects used community stakeholder input to develop programs that support social and problem solving skill development for students as well as a positive school climate. Drs. Sullivan, Sutherland, and Leff shared their insights from doing this type of work and collectively emphasized the importance of creating partnerships with schools and attending to the unique strengths and needs of each school.

What are some key elements of developing school-wide bullying prevention programs?

Drs. Sullivan and Sutherland: One key element is to work with administrators, teachers, and other school staff to understand school dynamics that foster prosocial behavior and those that may place students at risk for exposure to bullying behaviors (e.g., places in the school such as stairwells or bathrooms). Another is to have a strong school committee to assist with developing the program in order to maximize the relevance and meaningfulness of the interventions for students and school staff.

Dr. Leff: It is important to understand how the school has tried to address problems such as bullying in the past, as this provides important information about how to work best with formal and informal school leaders, and how your program can complement successful efforts already in place or support in areas that have been challenging. 

What are some keys to successful implementation?

Drs. Sullivan and Sutherland: One key to successful implementation is to monitor implementation progress via the collection of fidelity data, including data on student engagement (which schools are very interested in) and share these data with teachers and other staff, both to reinforce strong aspects of implementation as well as to highlight areas that need improvement. Another is to successfully engage with administrators; the more involved and supportive they are, the more successful implementation will be.

Dr. Leff:

  1. Establishing buy in from the principal, teachers, lunch-recess supervisors, and students.
  2. Developing internal champions within schools to help promote the program and speak to other teachers about the importance of the work.
  3. Discussing how to make programs sustainable is a conversation that needs to occur from day one.

What are the challenges to doing these types of school-wide interventions?

Drs. Sullivan and Sutherland: School-wide interventions are complex and ensuring that each component (individual, classroom, and school) is working well takes considerable effort.

Dr. Leff: These programs can be difficult to implement due to competing demands such as scores on state-wide and national testing. One of the main strategies is to help teachers understand how programs such as ours are able to improve the classroom teaching climate and thereby support academic and social-emotional functioning of the students. 

What have you learned from doing this type of work?

Drs. Sullivan and Sutherland: It’s a two-way street - we as researchers need to work hard to develop trust with the teachers, administrators and school staff, supporting them at every turn which can result in the long-term in a win-win for all parties.

Dr. Leff: One of the biggest lessons learned during the P4P has been how much the partnership between a school and team impacts the success of the program.

These two research studies are ongoing. As study results become available, we will learn whether these innovative interventions show promise for reducing incidents of bullying and improving students' achievement in school. Stay tuned!

Comments? Questions? Please send to IESResearch@ed.gov.

NCSER Celebrates Down Syndrome Awareness Month

By Kristen Rhoads, Education Program Specialist, Office of Special Education Programs

October is Down Syndrome Awareness Month – a time to celebrate what makes individuals with Down syndrome wonderful.  
 
According to the Centers for Disease Control and Prevention, Down syndrome occurs in 1 out of approximately every 700 births, with about 6000 babies born with Down syndrome in the United States each year. It is a lifelong, genetic disorder caused by a full or partial third copy of the 21st chromosome.  Individuals with Down syndrome typically demonstrate profiles of relative strengths in visuospatial processing,  social skills, and receptive language and needs for support in many areas including expressive language, motor, and cognitive skills.  They may also have other health-related issues, including most commonly: hearing loss, ear infections, sleep apnea, eye diseases, and heart defects. With services, supports, and high expectations for performance, many individuals with Down syndrome earn high school diplomas, participate in post-secondary education, live independently, and become valuable contributors to society.  
 
In celebration of Down Syndrome Awareness Month, we asked Dr. Stephen Camarata, Professor in the Department of Hearing and Speech Sciences at Vanderbilt University, for his thoughts on educating children with Down syndrome and potential directions for research. Dr. Camarata is the Co-Investigator for a NCSER-funded grant to evaluate the efficacy of interventions designed to produce speech accuracy and comprehensibility of elementary school students with Down syndrome.

What should families keep in mind when their child is initially diagnosed with Down syndrome?

In our Down Syndrome Clinic here at Vanderbilt University, there is an initial "burst" of medical activity when the family first learns that their child has Down syndrome. Families spend time making sure that the child’s basic health needs are met and have a lot of appointments doing imaging, surgery, and so on. Then, the family settles in as the Down syndrome unfolds. The child’s development –talking and communication, education, and behavior - all become immediate and long term foci.

Are there common misconceptions about individuals with Down syndrome?

A BIG problem is that there is a myth of a learning shelf life or a mythical critical period for learning.  All too often this means that educators quit trying to teach academic skills to people with Down syndrome when they reach the age of 10 or 12. Tragically, this can mean pushing children through custodial care with minimal academic content instruction until they "age out" of educational support.
 
Another important consideration that I sometimes see is that people have low expectations and, therefore, underestimate the learning abilities of a child with Down syndrome. In a sense, they set the bar "too low” or do not provide meaningful learning opportunities. Therefore, educators may inadvertently prevent a child with Down syndrome from reaching his or her potential. Down syndrome is highly variable, so it is important to provide multiple types of opportunities and let the child show you how much and how fast he or she can learn.

Are there areas of research or practice that you think require more attention?

With regard to key research areas, my own recommendations are to:

  1. Examine further the benefits of inclusion – for both a child with Down syndrome and his or her peers
  2. Develop and evaluate interventions or strategies that improve communication, speech, language, social and literacy skills, especially reading comprehension.
  3. Investigate learning in adolescence and develop and evaluate interventions that optimize academic and transition outcomes in middle school and beyond.
  4. Examine strategies to improve parent and family support. All of the terrific things that we have learned for training parents of children with Autism Spectrum Disorders may also work for families that include a child with Down syndrome.  More research in this area is needed. 

Visit our website, for more information about the research that NCSER funds.

Questions? Comments? Please send them to IESResearch@ed.gov.

IES Honors Statistician Nathan VanHoudnos as Outstanding Predoctoral Fellow

By Phill Gagne and Katina Stapleton, NCER Program Officers

Each year, IES recognizes an outstanding fellow from its Predoctoral Interdisciplinary Research Training Programs in the Education Sciences for academic accomplishments and contributions to education research. The 2014 winner, Dr. Nathan VanHoudnos completed his Ph.D. at Carnegie Mellon University and wrote his dissertation on the efficacy of the Hedges Correction for unmodeled clustering. Nathan is currently a postdoctoral fellow at Northwestern University. In this blog, Nathan provides insights on becoming an education researcher and on research study design. 

How did you become interested in education research?

I was born into it. Before he retired, my father was the Director of Research for the Illinois Education Association. Additionally, my grandparents on my mother's side were both teachers. 

 

As a statistician, how do you explain the relevance of your research to education practitioners and policy-makers?

I appeal to the crucial role biostatisticians play in the progress of medical research. Doctors and medical researchers are able to devote their entire intellectual capacity towards the development of new treatments, while biostatisticians are able to think deeply about both how to test these treatments empirically and how to combine the results of many such studies into actionable recommendations for practitioners and policy makers.  I aim to be the education sciences analogue of a biostatistician. Specifically, someone whose career success is decided on (i) the technical merits of the new methodology I have developed and (ii) the usefulness of my new methodology to the field. 

Your research on the Hedges correction suggests that many education researchers mis-specify their analyses for clustered designs. What advice would you give researchers on selecting the right analyses for clustered designs? 

My advice is to focus on the design of the study. If the design is wrong, then the analysis that matches the design will fail, and it is likely that no re-analysis of the collected data will be able to recover from the initial mistake. For example, a common design error is randomizing teachers to experimental conditions, but then assuming that how the school registrar assigned students to classes was equivalent to the experimenter randomizing students to classes. This assumption is false. Registrar based student assignment is a kind of group based, or clustered, random assignment. If this error is not caught at the design stage, the study will necessarily be under powered because the sample size calculations will be off. If the error is not caught at the publication stage, the hypothesis test for the treatment effect will be anti-conservative, i.e. even if the treatment effect is truly zero, the test statistic is still likely to be (incorrectly!) statistically significant. The error will, however, be caught if the What Works Clearinghouse decides to review the study. Their application of the Hedges correction, however, will not fix the design problem. The corrected test statistic will, at best, have low power, just like a re-analysis of the data would. At worst, the corrected test statistic can have nearly zero power. There is no escape from a design error. 


To give a bit of further, perhaps self-serving advice, I would also suggest engaging your local statistician as a collaborator. People like me are always looking to get involved in substantively interesting projects, especially if we can get involved at the planning stage of the project. Additionally, this division of labor is often better for everyone: the statistician gets to focus on interesting methodological challenges and the education researcher gets to focus on the substantive portion of the research. 

How has being an IES predoc and now an IES postdoc helped your development as a researcher?

This is a bit like the joke where one fish asks another "How is the water today?" The other fish responds "What's water?" 

I came to Carnegie Mellon for the joint Ph.D. in Statistics and Public Policy, in part, because the IES predoc program there, the Program for Interdisciplinary Education Research (PIER), would both fund me to become and train me to become an education researcher. The PIER program shaped my entire graduate career. David Klahr (PIER Director) gave me grounding in the education sciences. Brian Junker (PIER Steering committee) taught me how to be both methodologically rigorous and yet still accessible to applied researchers. Sharon Carver (PIER co-Director), who runs the CMU lab school, built in a formal reflection process for the "Field Base Experience" portion of our PIER training. That essay, was, perhaps, the most cathartic thing I have ever written in that it helped to set me on my career path as a statistician who aims to focus on education research. Joel Greenhouse (affiliated PIER faculty), who is himself a biostatistician, chaired my thesis committee. It was his example that refined the direction of my career: I wish to be the education sciences analogue of a biostatistician. 

The IES postdoc program at Northwestern University, where I am advised by Larry Hedges, has been very different. Postdoctoral training is necessarily quite different from graduate school. One thread is common, however, the methodology I develop must be useful to applied education researchers. Larry is, as one might suppose, quite good at focusing my attention on where I need to make technical improvements to my work, but also how I might better communicate my technical results and make them accessible to applied researchers. After only a year at Northwestern, I have grown considerably in both my technical and communication skills.

What career advice would you give to young researchers?

Pick good mentors and heed their advice. To the extent that I am successful, I credit the advice and training of my mentors at Carnegie Mellon and Northwestern. 


Comments? Questions? Please write to us at IESResearch@ed.gov.

Where Are They Now? A Q&A With the Creators of EcoMUVE – A Virtual Environment for Middle School Science

Where Are They Now? showcases completed IES research projects. The feature describes the IES project and research findings, and updates the progress since IES project completion.

By Ed Metz, NCER Program Officer

In this inaugural Where Are They Now? feature, we take a look back at a 2008 grant to researchers at Harvard University for the development of EcoMUVE.

EcoMUVE uses Multi-User Virtual Environments (MUVEs), which have the look and feel of video games, to help middle school students gain a deeper understanding of ecosystems, scientific inquiry, and causal patterns. The MUVEs recreate authentic ecological settings within which students explore and collect information. Students work individually at their computers and collaborate in teams within the virtual world. EcoMUVE includes two modules, Pond and Forest; each module is a two-week inquiry-based ecosystems curriculum. EcoMUVE received the First Place award in the Interactive and Immersive Learning Category at the 2011 Association for Educational Communications and Technology conference, and has received follow-on support from the National Science Foundation and Qualcomm Wireless. 

In this blog, we catch up with two of the researchers who led the development of EcoMUVE, Chris Dede and Shari Metcalf, to look back at their IES project and to learn about recent developments.

How and when did the idea to develop a virtual environment for science learning come about?

Chris Dede’s prior research with the River City project looked at supporting student inquiry using immersive exploration in a virtual world. Meanwhile, Harvard Professor Tina Grotzer was developing ways to support students in understanding complex causality in ecosystems. They worked together on a grant proposal to IES to combine their interests.

How does a virtual environment provide meaningful learning opportunities that otherwise might not be possible?

Ecosystems are complex systems shaped by relationships that often happen at microscopic levels, at a distance, and over long periods of time. Immersion in virtual environments can transform the learning experience by situating the learner in a rich simulated context in which new visualization opportunities are possible – e.g., zooming in to the microscopic level, or traveling to different points in time.

Students start to get a feel for the ecosystem and its relationships through tacit sensory clues. It is an uphill walk from the pond to the housing development, and students can walk down along a drainage ditch and through the pipe where runoff flows into the pond. The pond becomes noticeably greenish during the algae bloom. 

Students can experience turbidity directly by walking under the water of the pond and seeing how murky it looks on different days.

 

What was an unexpected outcome of the development process?

The types of “big data” about motivation and learning for each student that EcoMUVE can generate include: time-stamped logfiles of movements and interactions in the virtual world, chat-logs of utterances, and tables of data collected and shared. Other digital tools can provide data from concept maps that chart the flow of energy through the ecosystem and that document each student team’s assertions about its systemic causal relationships, with adduced supporting evidence. Using Go-Pro cameras, students’ collaborative behaviors outside of digital media can be documented. We would like to use this data to provide near-real time feedback to students and teacher, through various forms of visualization.

What were your main research findings from the IES development project?

After using EcoMUVE, students showed gains in learning of science content, and also improvements in their attitudes towards science, particularly in the beliefs they were capable and interested in being scientists. Teachers felt that the curriculum was feasible, well-aligned with standards, and supported student engagement and learning of science content, complex causality, and inquiry, and had multiple advantages over a similar non-MUVE curriculum. A study that looked at student motivation found that, while at first students were most enthusiastic about the 3D virtual world and game-like environment, over time their engagement centered on the inquiry-based pedagogy and the collaborative problem-solving.  Gains were also found in students’ complex causal reasoning about non-obvious causes; distant drivers of ecosystems dynamics and the system parameters; and processes, steady states and change over time.

How has the EcoMUVE project proceeded in recent years since the IES research project ended?  

Beginning in May, 2012, we’ve been pleased to be able to offer a standalone version of the EcoMUVE software for download through a free license from Harvard University. As of January, 2015, over 1,200 users have registered with the website. The EcoMUVE project receives e-mail inquiries almost every week from educators who are interested in the curriculum. In some cases, whole districts have adopted the EcoMUVE curriculum, including Cambridge, MA, and Peoria, AZ.

Internationally, researchers at the University of Hong Kong have been working with Harvard University to use EcoMUVE for professional development, to help teachers understand how to use scientific investigations as learning activities for students. Other collaborators include Sydney University, and Aalborg University in Copenhagen.

Looking ahead, what does the future hold for EcoMUVE?

We continue to make EcoMUVE available for download from our new website, http://ecolearn.gse.harvard.edu. We have been extending our research to develop EcoMOBILE, an extension of the EcoMUVE curriculum that blends immersive virtual environments with the use of mobile technologies during field trips to real ecosystems for observations and data collection. EcoMOBILE is funded by the National Science Foundation (NSF) and Qualcomm’s Wireless Reach Initiative. We have also just started a new research project, EcoXPT, also funded through NSF, designed to work alongside EcoMUVE to support experiment-based inquiry in immersive virtual environments.

Questions? Comments? Please send them to us at IESResearch@ed.gov.

About the Interviewees

Shari J. Metcalf is Project Director of the EcoMUVE project at the Harvard Graduate School of Education. She holds SB and SM degrees from MIT, and a PhD from the University of Michigan, where she designed and developed Model-It, a software tool for students building models of dynamic systems. Her professional focus is the design of educational software tools, and in particular on using modeling, simulation, and virtual immersive environments to support inquiry-based science learning.

Chris Dede is the Timothy E. Wirth Professor in Learning Technologies at Harvard’s Graduate School of Education.  Chris was the Principal Investigator of the EcoMUVE project. His fields of scholarship include emerging technologies, policy, and leadership.  His research includes grants from NSF, IES, and the Gates Foundation to design and study immersive simulations, transformed social interactions, and online professional development.