There are many NAEP-related presentations and training opportunities at the annual meetings of the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME). Below is a list of sessions from the AERA and NCME programs; you will find more detailed abstracts through their websites. At the conference, please check for schedule updates.
Register for training sessions when you register for the meeting; there is a charge. Direct questions about the professional development and training courses to profdevel@aera.net.
TRAINING SESSIONS:
Thursday, 4/07
Saturday, 4/09
Monday, 4/11
PAPER AND POSTER SESSIONS:
Friday, 4/08
Saturday, 4/09
Sunday, 4/10
Monday, 4/11
---------------------------------
TRAINING SESSIONS:
Thursday, 4/07
9:00 a.m.–5:00 p.m. New Orleans Marriott / La Galerie 4
PDC07: Psychometrics Behind National Assessment of Educational Progress: Understanding and Analyzing NAEP Data
Director and Instructors: Emmanuel Sikali, Enis Dogan, Andrew J. Kolstad
The goal of this course is to introduce researchers to the National Assessment of Educational Progress (NAEP) and the AM Statistical Software analysis tool, which make these rich databases more accessible to researchers than it has ever been before. AM is a free statistical software package for analyzing data from complex samples, especially large-scale assessments. This course will introduce users to the psychometric and sampling design of NAEP. Using the NAEP data file, instructors will introduce several data analysis strategies, including the Marginal maximum likelihood approach to computing scale scores, use of sampling weights, and variance estimation procedures.
Fee: $95
1:00 p.m.–5:00 p.m. Hotel Monteleone / La Nouvelle Orleans West
PDC16: Using National Assessment for Educational Progress Data on the Web for Educational Policy Research
Session Directors and Instructors: Debra Kline, Catherine S. Trapani, Emmanuel Sikali
This course is for researchers interested in National Assessment of Educational Progress (NAEP) data and focuses on using the NAEP Data Explorer Web tool to examine the wealth of assessment data collected since 1990. The course provides hands-on learning and active participation. The participants will be guided through an examination of the data, with emphases on (1) the relationships between student performance and teacher and school characteristics and (2) using NAEP data to augment or confirm other education research findings. Participants will have the opportunity to work independently according to their interests. This course focuses on the 2009 math and reading assessments and the most recent assessments of science and writing. Laptop computers with wireless internet cards are required.
Fee $95
8:00 a.m.–12:00 p.m. Hotel Monteleone / La Nouvelle Orleans East
PDC 24: Data Analysis on Simple Random Samples and National Center for Educational Statistics Complex Samples Using the R Software
Director: Emmanuel Sikali
The goal of this course is to introduce educational researchers to the R software. This software is free and open source, and contains several packages for analyzing simple random and complex sample data. This course will introduce researchers to statistical data cleaning, recoding, data analysis strategies. Additionally participants will learn about complex sample design, the use of sampling weights, variance estimation procedures, and the appropriate R packages to perform analysis with such samples.
Fee: $50
1:00 p.m.–5:00 p.m. Hotel Monteleone / Bonnet Carre
PDC25: Accessing and Analyzing High School Transcript Study Data for Inspired Educational Research Purposes
Directors and Instructor: Janis D. Brown, Jennifer Laird, Stephen E. Roey , Robert Colby Perkins
This course will provide graduate students, faculty, and researchers information on how to access and analyze the National Assessment of Educational Progress (NAEP) High School Transcript Study (HSTS) data. Topics covered by this course will include 1) the HSTS survey design; 2) technical issues in the proper use and handling of sampling weights and plausible values; and 3) a discussion and demonstration of current specialized software for accessing and analyzing HSTS data. The course will include extensive demonstrations, independent exercises, and group discussions. A laptop computer with a wireless card is required for full participation.
Fee $95
PAPER AND POSTER SESSIONS:
2:15 p.m.–3:45 p.m. Doubletree / Rosedown A
Symposium: Communicating Assessment Results to Particular Audiences
One of 5 papers:
Next Steps in Improvements to Student Score Reporting: Emerging Methodologies and Evaluative Criteria
Ronald K. Hambleton, April L. Zenisky
Testing practices in education have advanced considerably in recent years through the introduction of item response theory models, generalizability theory, automated test assembly and scoring, new test designs such as computer-adaptive testing, and the emergence of new item types to assess higher-level thinking skills. At the same time, methods for reporting test scores and diagnostic information to students, the culmination of the testing process, have remained largely understudied and undervalued as a problem in educational assessment until recently. This was most unfortunate because of the large amount of evidence suggesting that students and other score users such as parents, teachers, school administrators, policy-makers, and the media, were often confused by the meaning of test scores and students, parents, and their teachers were often disappointed by the limited amount of diagnostic information they received from hours of testing. We are pleased to report that we believe considerable progress on the topic of score reporting has been made in recent years. College Board has prepared an enhanced way to report SAT scores. NAEP is setting the national standard for providing quality group score reports on the internet. Many of the states have adopted more clear score reports, improved the quality of the information they are reporting, and these same states are attempting to provide diagnostic information that meet measurement standards of quality. With lots of new work to guide our research, the purposes of this paper include: (1) reviewing and providing examples of emerging methodologies for improving score report designs, and, (2) evaluative criteria for use with student score reports. The emerging methodologies include “think-aloud” studies, focus groups, experimental research, tryouts, and qualitative reviews. The evaluative criteria we will present have evolved from our own research over the last ten years working with College Board, NAEP, many state departments of education, and several credentialing agencies.
4:05 p.m.–6:05 p.m. Sheraton / Oakley H
NAEP Studies SIG Session: Achievement Gap, Construct Irrelevant Variance, Socioeconomic Status, and Background Variables in NAEP and Course-Taking Trends
Chair: Cadell Hemphill Discussant: Rolf K. Blank
Mind the Gaps: States, Race, and Income
Gregory J. Marchant , Sharon E. Paulson
A Lack of Construct Irrelevant Variance in National Assessment of Educational Progress Math Items for Students With Learning Disabilities (So Far)
Luke S. Duesbery
Deciphering Socioeconomic Status: Understanding the Association Between SES and Student Background Variables in the National Assessment of Educational Progress
Burhan Ogut, Salvador Rivas
Trends in Noncore Academic Courses: 1990 to 2009
Janis D. Brown, Jennifer Laird
This paper session presents four papers on achievement gap, construct irrelevant variance for students with learning disabilities, association of SES and other background variables, and course taking trends. The first paper explores achievement gaps by race/ethnicity and income in the National Assessment of Educational Progress (NAEP) using multiple regression analysis and a modified “value-added” approach across states. The second study analyzes student performance from NAEP to determine the extent to which students with disabilities experience question level discrimination associated with graphical literacy access skills. The third paper evaluates the associations among socioeconomic status (SES) and other proxy measures of SES in NAEP using SES measures from ECLS-K, by utilizing a statistical link between two assessments. The fourth paper examines high school graduates’ course taking trends in non-core academic subjects using estimates from the 1990, 2000, 2005 and 2009 High School Transcript Studies (HSTS) associated with NAEP.
4:05 p.m.–5:35 p.m. Sheraton / Grand Ballroom A
Roundtable Session 7: Accommodations Policy for Large-Scale Assessment: What We Know About Implementation at the Local Level
One of 4 papers:
National Assessment of Educational Progress (NAEP) 2008 Grade 4 Inclusion Block Study
Lizanne DeStefano
The NAEP Validity Studies Panel and other groups have been interested in the use of modified blocks as a means of improving measurement at the lower levels of the NAEP scale and increasing the accessibility and validity of NAEP to the groups of students now excluded from the assessment. The aim of including one or more “inclusion blocks” would not be to make NAEP easier, but to improve measurement at the lower end of the performance continuum by including more items that provide information about those students’ abilities and skills. Increased precision at the lower levels represents an important validity issue regarding the use of NAEP as a means of benchmarking and interpreting both status and change in student assessment results over time. The study to create and pilot “inclusion” blocks was conducted in conjunction with the 2008 grade 4 pilot field test and involved: 1: Developing a definition of an “Easy Block” 2: Implementing a process for constructing Inclusion Blocks that are consistent the NAEP Frameworks: 3: Administering Inclusion Blocks in the 2008 Grade 4 pilot/field test Two inclusion blocks developed and tested in the cognitive labs were field tested in two ways: 1. a random sample of students (N=671) were administered booklets composed of two math inclusion booklets; and 2. a sample of students who would otherwise be excluded from NAEP (N=41) were administered an “inclusion booklet” consisting of two inclusion blocks. Results Random Sample Study 1. The average percentage of items omitted and average percentage of items not reached is very low for total group and all subgroups. 2. The inclusion blocks were significantly easier than the 2007 operational blocks with average percentage correct ranging from 80.81 % to 75.76%. Average p-values for operational blocks were 57% and 50% respectively. 3. The range of p-values for the operational sample was .30-.95. The range for the inclusion sample was .45-.95. Excluded Student Study 1. Excluded student sample obtained average percentage correct of between 51% and 57%. These are students who would not have taken NAEP under typical circumstances. 2. Average percentage items omitted or not reached were low for this group as well. 3. Performance on Extended Constructed Response Items was lower than Multiple Choice and Short Constructed Response Items for this group. Implications 1. Results support the use of the process described above to create inclusion blocks that are consistent with the NAEP Framework and produce higher average percentage correct than traditional blocks. More research is needed to determine the extent to which incorporation of inclusion booklets into the regular administration of NAEP increases precision at the lower end of the performance continuum. 2. Availability of an inclusion block increased the participation of traditionally excluded students in NAEP Mathematics. 3. Application of Item Modification Guidelines and Expert Review aspects of the process described above in the general NAEP item development process could result in items that are more accessible and reduce construct irrelevant variance for NAEP as a whole.
12:25 p.m.–1:55 p.m. Westin / Magnolia Ballroom I, C3
NCME Coordinated Session: To Break Trend or Not, That is the Question
Organizers/Moderator: Young Yee Kim, George Bohrnstedt
Discussants: Peggy Carr, Lorrie Shepard
Principles and Research Studies to Guide Decisions About Whether to Maintain or Break Trend When a New Assessment Framework is Introduced
Jeffrey Nellhaus, Peter Behuniak, Frances Stancavage
To Break or Continue Trend: The Role of Content Comparisons of Old and New Assessment Frameworks
Kim Gattis, Young Yee Kim, George Bohrnstedt, Peter Ward
To Break or Continue Trend: The Role of Psychometric Studies of Old and New Items
Rebecca Moran, Meng Wu
To Break or Continue Trend: The Role of Cross-Assessment Item Pool Variation
Young Yee Kim, Sharyn Rosenberg, Teresa Neidorf
2:15 p.m.–3:45 p.m. Westin / Azalea Ballroom II, D2
NCME Session: College Readiness, Career Readiness: Same or Different?
One of 3 papers:
Overview of NAGB Studies Designed to Compare Workplace and College Preparedness
Susan Loomis
4:05 p.m.–6:05 p.m. Westin / Azalea Ballroom II, E2
NCME Invited Symposium: Opportunities and Challenges to Meeting Diverse Needs Through Technology-Based Testing
One of 5 papers:
Reaching Diverse Learners Using Technology-Based Assessment: A National Perspective
Holly Spurlock
4:05 p.m.–5:35 p.m. Sheraton / Grand Ballroom D
Roundtable Session 22: Closing Achievement Gaps Between Subgroups and School Accountability
One of 5 papers:
Closing the Achievement Gap? Analyzing Change Since No Child Left Behind Using State Assessments and the National Assessment of Educational Progress
Rolf K. Blank
This paper presents findings from analysis of change in assessment scores by state and by population groups using both state assessment data and NAEP data. We provide suggestions on how these data sets can be used to address common questions raised by decision-makers, educators, and the public, and we also demonstrate methods of displaying analysis of closing the gap. We recommend that these kinds of trends data become more prominently displayed and reported in district, school, and state report cards and online reporting systems.
8:15 a.m.– 9:45 a.m. Doubletree / Madewood A
Symposium: Reading Researchers in Search of Common Ground: The Expert Study Revisited
One of 4 papers:
A Focus on Past and Current National Assessment of Educational Progress Reading Data
Jay R. Campbell
This symposium revisits the original Expert Study and its significance, presents the results of a recent follow-up study examining what leading experts identify as the most important literacy research over the past ten years, positive and/or negative, and how these findings should (or could) inform contexts and practices of reading instruction in classrooms. Panelists share their views, findings, and theories based on the use and power of the Delphi approach to problem solving, past and current NAEP reading data, Cognitive Flexibility Theory, and theory from the “Radical Middle”. Finally, attendees are encouraged to interact with the expert panel and share their perspectives regarding the most important literacy research over the past ten years during the culminating discussion.
10:35 a.m.–12:05 p.m. Sheraton / Salon 824
NAEP Studies SIG Session: Association of Student Achievement With Background Variables, Engagement, and Motivation
Chair: Samantha S. Burg Discussant: Andreas H. Oranje
New Item Models for Engagement: Simultaneous Identification of Engagement and Adjustment of Reporting Group Differences
Murray Aitkin, Irit Aitkin
Student/Teacher Factors Associated With the Math Achievement of American Indian/Alaska Native Students Using 2009 National Assessment of Educational Progress/National Indian Education Study (NAEP/NIES) Data
Chun-Wei (Kevin) Huang, Sharon Nelson-Barber, Elise Trumbull, Ursula M. Sexton
An Investigation of Student Responses to Questions of Effort and Motivation on National Assessment of Educational Progress: Multiple Regression and Internal Consistency Reliability Comparisons
Carina M. McCormick
This paper session presents three papers exploring the importance of background variables, engagement, and motivation on student achievement. The first paper presents an evaluation of new item models to represent engagement of students in the NAEP task in terms of the relationship between student achievement and background variables. The second paper explores teacher- and student-level factors that are associated with the achievement of AI/AN students on the mathematics assessment in NAEP. The third paper explores if low test taker motivation for the NAEP potentially undermines interpretation of results.
12:25 p.m.–1:55 p.m. Westin / Magnolia Ballroom III, G4
NCME Paper Session: Estimating Parameters of Multidimensional Item Response Theory Models
One of 5 papers:
Unidimensional and Composite Scaling Comparisons of NAEP
Meng Wu, Adrienne Sgammato, Yue Jia
2:15 p.m.–3:45 p.m. Westin / Azalea I
NCME Session: Possibilities and Limitations in Drawing Inferences from Large Scale Survey Assessments
Presenters: Jacquelin W. Cocke, David Kaplan, Eckhard J. Klieme, Jack Buckley
Discussant: Andreas H. Oranje
4:05 p.m.– 6:05 p.m. Westin / Azalea Ballroom II, I2
NCME Paper Session: Scoring and Score Reporting
One of 5 papers:
Performance of Reliability Measures for NAEP
Adrienne Sgammato
6:15 p.m.–7:45 p.m. Sheraton / Salon 824
NAEP Studies SIG Business Meeting
12:25 p.m.–1:55 p.m. Sheraton / Grand Ballroom E
Roundtable Session 44: Test Accommodation Research: Item Difficulty, Test Accessibility, Policies, and Perceptions
One of 5 papers:
Effects of Linguistic Complexity and Accommodations on National Assessment of Educational Progress Item Difficulty for Students With Learning Disabilities
Stephanie W. Cawthon, Susan Natasha Beretvas, Alyssa Kaye, L. Leland Lockhart
Research has indicated that Linguistic Complexity (LC) may confound results on standardized assessments. Previous research has focused on the English Language Learner population, yet very little is known about LC and test performance for students with learning disabilities (SLD). This study entailed an analysis of NAEP mathematics and reading items. For both mathematics and reading, the higher an item’s LC, the more difficult it was for SLDs. After controlling for differences in accommodated versus unaccommodated scores, LC was not a significant predictor of mathematics items’ difficulties although it remained a significant predictor for reading items. No significant LC by accommodations interactions were found, implying accommodations did not reduce LC’s effects on item difficulties. Cross-validation analyses supported these results.
12:25 p.m.–1:55 p.m. Westin / Azalea Ballroom II, L2
NCME Paper Session: Issues With Multistage Testing
One of 4 papers:
Multi-Stage Testing in Educational Survey Assessments
Xueli Xu, Andreas Orange, Emmanuel Sikali, Ed Kulick
