Here is a list of NAEP-related sessions excerpted from the preliminary conference program (as of April 11) on the Council of Chief State School Officers (CCSSO) NCSA web pages. Consult the program for pre- and post-sessions.
When you arrive at the conference, please check for any schedule changes. All sessions will be held in The Peabody Orlando.
Monday, 6/20/2011
Tuesday, 6/21/2011
Wednesday, 6/22/2011
In an increasingly interconnected world, international assessments such as PIRLS, PISA, and TIMSS are receiving more attention. We will discuss the purpose, structure, and reporting of these assessments as compared to NAEP. We will also discuss the NAEP-TIMSS International Linking Study and what this study means in the context of our states and in policy discussions.
Education stakeholders at the state and national levels are calling for international benchmarking to determine how the achievement of children in the United States compares to that of children around the world and to ensure that students are prepared for the increasingly global job market. PIRLS, PISA, and TIMSS, assessments that provide statistical data for cross-national comparisons, are thus receiving more attention than ever. The most recent PISA release resulted in an increased clamor for international benchmarking of state standards and student achievement. In 2011, the concurrence of the NAEP and TIMSS assessment administrations will allow for a special study linking state-level mathematics and science results from NAEP with the TIMSS mathematics and science scales and the TIMSS scores of participating countries from across the globe. PIRLS is also being conducted this year, and recruitment efforts benefit from this growing focus on international assessments. NAEP remains an important measure of the country’s educational progress and has become an integral part of some state assessment systems. With international assessments securing a place alongside NAEP in state assessment systems, we will discuss how NAEP, PIRLS, PISA, and TIMSS fit together and their relevance at the state level.
First, we will offer an overview of these assessments, comparing and contrasting NAEP, PIRLS, PISA, and TIMSS. The presenters will discuss the different structures of these assessments, how their results are being used by various audiences, and their relevance in the state context. Panelists from Arkansas, Iowa, Kentucky, and Minnesota will share perspectives of the role of these international assessments in each of their state assessment systems. Kate Beattie, the Minnesota NAEP State Coordinator (NSC), will discuss why Minnesota opted to participate in TIMSS in 2007 and how results were shared within the education and business communities. Pam Byrd of Arkansas, Dianne Chadwick of Iowa, and Mark DeCandia of Kentucky, will share how their states plan to use projected TIMSS scores through the NAEP-TIMSS linking study. Laura Egan will conclude with a summary of salient policy discussions related to international benchmarking, the relevance of international assessments to state assessment systems and NAEP, and the role of these assessments in measuring academic achievement in this changing world.
Lead Presenter: Laura Egan
Co-presenters: Kate Beattie, Pam Byrd, Dianne Chadwick, and Mark DeCandia
10:00 a.m.–11:30 p.m.; Plaza International I
Stretching NAEP--but in Which Directions?
During their May 2010 quarterly meeting, members of the National Assessment Governing Board (NAGB) received this challenge from Chairman David Driscoll regarding NAEP: “The question for the Board is how do we stretch ourselves and start to become bolder?” During this interactive, audience-participation session, NAGB representatives will describe their views regarding “stretch” possibilities for NAEP and solicit suggestions from attendees.
Expressing his concerns about the current status of U.S. education, Driscoll continued, “It is sometimes necessary to join the public discussion when there is a sense of urgency about what is happening around the country in education.” During follow-up discussions, Board members have continued to explore additional ways in which NAEP might contribute educationally without marring its traditional functions. But NAGB also plans to solicit suggestions from other concerned constituencies. The proposed session is deliberately crafted to function as an outreach effort by NAGB—an effort intended to garner reactions to a number of NAGB-generated options and also to secure suggestions from those attending the Orlando conference.
This three-phase session is deliberately designed to secure useful advice for NAGB from NCSA attendees regarding potential “stretch” possibilities for NAEP.
Phase one of the session, moderated by NAGB Chairman David Driscoll, will feature five representatives of NAGB who will each present one or, perhaps, two ways in which NAEP, or the many resources associated with it, might be employed to stimulate educational improvements in the U.S. Four Board members and the Executive Director of NAGB will make these presentations of no more than five minutes apiece. At that point, members of the audience will arrange themselves in small groups to (1) discuss the suggestions they just heard and (2) identify other potential “stretch” possibilities for NAEP. This second phase of the proposed session will last for 25-30 minutes. Finally, in the third phase, Chairman Driscoll will moderate an open discussion during which either individuals or subgroups from the audience can identify and defend the options they have been discussing. It is hoped that a lively exchange will take place among audience members and NAGB representatives.
A diverse group of four NAGB members will offer, independently, their views regarding whether NAEP’s role should be expanded and, if so, in which ways. The Board members are David Driscoll, a former state chief; Steven Paine, a current state chief; Louis Fabrizio, director of a state accountability program; and W. James Popham, an assessment specialist. Cornelia Orr, NAGB’s Executive Director, will be the final presenter.
Lead Presenter: David Driscoll
Co-presenters: Steven Paine, Lou Fabrizio, W. James Popham, and Cornelia Orr
8:30 a.m.–10:00 a.m.; Plaza International D
When You Wish Upon a Star: Computer-Based Assessments for NAEP
In the near future, all NAEP assessments will be administered via computer. This session will address promises and challenges in transitioning NAEP to a computer-based format. These innovative assessments will advance the field by offering tasks that tap higher-order skills, engage diverse student populations, and produce rich achievement data. Presenters will represent various perspectives, including those of policymakers, practitioners, and measurement experts.
During the last decade, NAEP has experimented with a variety of computer-based assessments in various subject areas. Beginning in 2011, NAEP begins a new era with its first completely computer-based assessment in writing. Other NAEP computer-based assessments are in the reporting and design phases including Science Interactive Computer Tasks given in 2009.
In 2014, the NAEP Technology and Engineering Literacy assessment (TEL) will be administered to a representative national sample of 8th graders. The symposium will explore the need for such an assessment, measurement goals, the overall design, and approaches to analysis. The approach, evidence-centered design (ECD) is new for NAEP. We will pay special attention to the approach and its effect on the overall design and development effort. The entire assessment will be computer-based and will include several innovative item types. Administrators, teachers, researchers, and the general public will be able to review and make use of the results to understand more about the state of 8th graders’ technological and engineering literacy at the national level to inform their work and policies.
The session presenters will provide an overview of lessons learned from the policy, practitioner, and measurement perspectives. These findings have implications for new systems of classroom, state, and national assessments in terms of student engagement, measurement of complex skills, and increased accessibility for special populations. Implications for 21st century skills will also be discussed in relation to content that students will need to compete in today’s global economy. Presenters will share examples of prototype tasks for audience discussion. Finally, the session will conclude with a look at challenges and strategies for scoring simulation-based assessments as well as further computer-based innovations for NAEP in the next decade.
Lead Presenter: Mary Crovo
Discussant: Randy Bennett
Moderator: Gina Broxterman
Co-presenters: Malcolm Bauer, Jay Campbell, and Dianne Walsh
10:30 a.m.–12:00 p.m.; Plaza International J
A Symposium on NAEP Grade 12 Preparedness Judgmental Standard Setting
Abstract:
Included in the priorities of the NAEP 12th Grade Preparedness Research is setting cut scores on the grade 12 reading and mathematics assessments indicating preparedness for college entry and entry in a job training course. In this session, the features of the judgmental standard setting process will be discussed and procedural results will be presented.
Presentations:
In 2009, the National Assessment Governing Board (NAGB) adopted a multifaceted program of research aimed at providing validity evidence for reporting preparedness on 12th grade NAEP. The program of research was formulated and recommended by a panel representing a variety of technical expertise related to preparedness for postsecondary education and job training. Judgmental standard-setting studies were recommended as one of several methods to identify points on the NAEP scale to reference preparedness for college credit-bearing, entry-level courses of the type that would fulfill general education requirements or preparedness for placement in a job training program for one of several occupational areas to be specified for this research. (NAGB, 2009) In 2011, 12 operational standard setting studies will be implemented for the 2009 grade 12 NAEP in reading and mathematics. In this symposium, the steps in this innovative set of judgmental standard-setting studies will be traced and some preliminary results will be presented.
The first two presentations will provide the context for NAGB's 12th grade preparedness research studies. First, the role of these studies in the Governing Board’s 12th Grade NAEP Preparedness Research program will be presented. Second, information on the meaning of preparedness in the NAEP context will be presented and compared with other definitions currently under consideration.
The third presentation will describe the process for developing performance standards used for the judgmental standard setting and highlight key procedural results. To develop preparedness performance standards, panelists will be assembled online for webinars with the goal of drafting performance standards that they will subsequently finalize during the on-site standard setting panel meetings.
The fourth presentation will focus on the judgmental standard setting process to set the cut scores using the Bookmark method. Several computer-based innovations of the method will be implemented in six judgmental standard setting workshops. A demonstration of some elements of the Computer Aided Bookmark (CAB) is included in the symposium. Preliminary results will also be presented.
Reference:
National Assessment Governing Board (2009). Making New Links, 12th Grade and Beyond: Technical Panel on 12th Grade Preparedness Research Final Report.
Lead Presenter: Susan Loomis
Discussant: Kevin Sweeney
Moderator: Stanley Rabinowitz
Co-presenters: David Conley, Luz Bay, and Jennae Bulat
