Skip Navigation
small NCES header image

NAEP Secondary Analysis Grant Abstracts

Read about projects funded in fiscal years 2001, 2000, 1999, 1998. Descriptions of NAEP secondary analysis projects funded in 2002 through 2007, and other NAEP research funded through the Institute of Education Sciences (IES), are in a database on the IES website; search for "NAEP." Return to the Funding Opportunities page on this website for additional research support sources.

Projects Funded in Fiscal Year 2001

Grantee: RAND Corporation
P. Investigator: David Grissmer
Project Title: Analyzing State NAEP Data to Address Educational Policy issues in K-12 Education

This project will expand earlier research by David Grissmer using NAEP state data by adding the 2000 fourth and eighth grade math data to the existing 1990 - 1998 trend line, and by including new variables such as Head Start participation, and descriptions of the assessment systems in the participating states to the models. The project will attempt to determine whether score gains across states apply to differing localities within those states; to determine whether some states are more successful than others in reducing the black-white score gap; and to determine whether level and targeting of resources by states is linked to gap reduction.

Publications:

Flanagan, Ann and David Grissmer, Tracking the Improvement in State Achievement Using NAEP Data, in Instructional and Performance Consequences of High Poverty Schooling , Michael Ross and George Bohrnstedt (eds), forthcoming

Grantee: LMP Associates, Inc.
P. Investigator: Lawrence Rudner
Project Title: Scoring Content Essays Using Bayesian Networks

This project will conduct basic research on the feasibility of using artificial intelligence systems based on Bayesian networks to evaluate and score extend response assessment items. If this approach proves feasible, programs could be developed which would reduce the cost and improve the speed and accuracy with which NAEP scores extended response items.

Grantee: University of Maryland
P. Investigator: Clare Von Secker
Project Title: Science Achievement in Social Contexts: An Alternative Method for Analysis of NAEP Data

This project will explore the compensatory impact of four classes of social and psychological "protective factors" on fourth, eighth, and twelfth grade achievement in science. Particular emphasis will be placed on attempting to determine the effect of these factors on disadvantaged students. The study hypothesizes that protective factors may help to clarify why some disadvantaged populations seem particularly vulnerable while others seem surprisingly resilient.

Grantee: University of Rochester
P. Investigator: Richard Niemi
Project Title: Components of Knowledge in the NAEP 1998 Civics Main and Trend Assessments

This project will conduct in depth analyses of the 1998 civics assessment data. The project intends to determine what specific aspects of fourth, eighth, and twelfth graders' knowledge of government, politics and civics have increased or decreased over the past decade and to identify variations in subgroup knowledge in each of the domains-content, skill, disposition, and context—specified in the 1998 civics framework.

Grantee: American Institutes for Research
P. Investigator: Daniel Sherman
Project Title: Application of Small Area Estimation Methods to NAEP

This project will apply small area estimation methods to NAEP to determine the extent to which these methods can be used to develop statistics for domains such as school districts, counties or other geographic areas for which sample sizes would otherwise be too small to create reliable statistics. A key outcome of the project will be an indication of how accurately achievement can be estimated in relatively small domains given available data and potential improvements in precision that could be obtained under alternative sampling schemes.

Grantee: Educational Testing Service
P. Investigator: Claudia Gentile
Project Title: Evaluating the creative in Creative Writing

This project will explore a method for evaluating the creative aspect of students' creative writing by analyzing the data from the 1998 NAEP Classroom-based Writing Study. While standards-based reform efforts have encouraged the teaching of creative writing, no rubrics exist to help define and evaluate the creative component of students' writing. This project will use the personal narratives, fictional stories and poems collected in the classroom writing study to develop rubrics that classroom teachers can use to respond to and encourage students' creative writing.

Grantee: Act, Inc.
P. Investigator: Matthew Schultz
Project Title: Describing Achievement Levels with Multiple Domain Scores

This project will characterize the NAEP eighth grade mathematics data in terms of multiple domain scores and reference these domain scores to the NAEP achievement levels. By defining domains in terms of both the categories of the NAEP mathematics framework and the instructional sequence of the content in the framework, the project will attempt to develop domain scores which allow educators and policy makers to identify instructionally relevant patterns of achievement in the NAEP data.

Back to Top

Projects Funded in Fiscal Year 2000

Grantee: Synectics for Management Decisions, Inc.
P. Investigator: Gary Huang
Project Title: Local School District Spending and Student Achievement: A synthetic analysis of the data from NAEP and CCD

This project will combine eighth grade mathematics data from three NAEP assessments with school district background information from CCD for the same years and examine the extent to which student achievement is related to districts' control over expenditures for instruction.

Publications:

Huang, G and Yu, B. District Fiscal Policy and Student Achievement: Evidence from Combined NAEP-CCD Data, Educational Policy Analysis Archives, 10(38), http://epaa.asu.edu/epaa/v10n38/.

Grantee: Datametrics Research, Inc.
P. Investigator: Donald Rubin
Project Title: Improving the Robustness of NAEP Statistical Methods

This project proposes to improve the statistical estimation of NAEP parameters by expanding the current NAEP imputation model to permit heterogeneous variances in key reporting subpopulations. The project will modify the EM algorithm utilized by NAEP and extend it to fit the expanded model. The validity of reporting based on the current NAEP model and the expanded model will be evaluated using both real and simulated assessment data.

Grantee: RAND Corporation
P. Investigator: David Grissmer
Project Title: Using the State NAEP Scores to Estimate and Explain the Pattern of Score Gains and Differences Across States by Racial/Ethnic Group for Urban, Suburban, and Rural Areas

This project will extend state comparison research conducted by the principal researcher by: (1) estimating score trends for racial/ethnic groups using the state-by-state NAEP data from 1990 through 2000, (2) estimating differences in scores for students from similar family backgrounds by state within racial/ethnic group, and (3) estimating the effect of educational resources on the achievement of students from different racial/ethnic backgrounds and from urban/suburban and rural communities.

Publications:

Flanagan, Ann and David Grissmer, The Role of Federal Resources in Closing the Achievement Gaps of Minority and Disadvantaged Students, in "Closing the Gap: Promising Approaches to Reducing the Achievement Gap." Thomas Loveless, (ed), Brookings, Wash D.C., 2002.

Grissmer, David and Ann Flanagan, Searching for Indirect Evidence for the Effects of State Reform, in Education, 2000, Diane Ravich (ed), Brookings, 2001.

Grissmer, David, Future Research Directions for Understanding the Relationship of Educational Resources to Educational Outcomes, in A Survey and Reassessment of Education Finance at the Start of the Millennium, Steve Chaikind and William Fowler, (eds), American Educational Finance Association, Larchmont, NY: Eye on Education.

Grissmer, David and Ann Flanagan, Moving Educational Research Toward Scientific Consensus, in Analytic Issues in Student Achievement, D. Grissmer and M. Ross, (eds), NCES, Department of Education, Wash DC, 2000.

Grantee: University of Pittsburgh
P. Investigator: Clement Stone
Project Title: Evaluation of a Wald Test for Assessing the Significance of a Fit Statistic Based on Posterior Expectations in Item Response Theory Models

In order to improve our ability to validate NAEP items, this project will evaluate a Wald test for determining the statistical significance of goodness of fit statistics which take into account the dependency that exists in distributions used to estimate NAEP item parameters. The project will use real and simulated NAEP data to the evaluate the performance of the Wald test, and will develop software for testing the fit of NAEP items to their estimation model using the Wald test to determine the statistical significance of those fit statistics.

Grantee: Advance Research & Data Analysis Center
P. Investigator: Jamal Abedi
Project Title: Assessment of NAEP Accommodation Strategies for Students with Limited English Proficiency

This project proposes to investigate six issues surrounding NAEP's first provision of accommodations for students with limited English proficiency who participate in the assessment. The issues include: (1) the effectiveness of NAEP's accommodations, (2) the validity of accommodated test results, (3) the differential impact of accommodations on students with different backgrounds, (4 ) the impact of linguistic complexity of the original test item on the effectiveness of accommodations, (5) the effect of teacher and school characteristics on the effectiveness of accommodations, and (6) the feasibility of implementing NAEP's accommodations in a full assessment setting.

Publications:

Abedi, J and Lord, C. (2001) The effects of Accommodations on the Assessment of Limited English Proficient (LEP) Students in NAEP. Center for the Study of Evaluation Technical Report No. 537.

Grantee: William Carey College
P. Investigator: Read Diket
Project Title: Implications of the 1997 NAEP Visual Arts Data for Policies Concerning Artistic Development in America's Schools and Communities

This project proposes an in-depth analysis of the data from the 1997 Arts Assessment in order to: (1) document constructs embedded in the visual arts data, (2) examine regional differences in the light of these constructs, (3) document variations on the constructs for students in the higher and lower quartiles of the responding and creating subscales, and (4) report relevant findings to local, regional and national decision makers in the arts education community.

Publications:

Diket, R. M. (2001). A factor analytic model of eighth-grade art learning: Secondary analysis of the NAEP Report Card in the visual arts. Studies in Art Education, 43(1), 5-17.

Diket, R. M., Burton, D., McCollister, S., & Sabol, F.R. (2000). Taking another look: Secondary analysis of the NAEP Report Card in the visual arts. Studies in Art Education, 41(3), 202-207.

Diket, R. M., Sabol, F.R., Burton, D., Thorpe, P., & Siegesmund, R. (Fall, 2002). What can we learn from the NAEP Arts? Translations: from Theory to Practice.

Sabol, F. R. (2001). Regional analysis of the NAEP. Studies in Art Education, 41(3).

Burton, D. (2001). Quartile analysis of the NAEP. Studies in Art Education, 41(3).

Siegesmund, R., Diket, R., & McColloch, S. (2001). Re-visioning the NAEP: Amending a performance assessment for middle school students. Studies in Art Education, 41(3), 45-56.

Grantee: University of Georgia
P. Investigator: Sue Sloop
Project Title: Impact of State Education Policy on Student Achievement: Evidence from the NAEP 1996 Mathematics State Assessment for Georgia and North Carolina

This project will use hierarchical linear models to identify factors that affect students' NAEP mathematics performance in a "low performing" Southern state, and study the differences these models highlight when applied to the NAEP mathematics data from a "high performing" Southern state with a similar student population.

Back to Top

Projects Funded in Fiscal Year 1999

Grantee: University of Massachusetts—Amherst
P. Investigator: Hariharan Swaminathan
Project Title: Impact of Errors in Item Parameter Estimates on the Estimation of Ability in NAEP

This project has two major components. It will: (1) investigate the impact of errors made in the estimation of item parameters on the accuracy of the proficiency score distributions used for reporting NAEP results, and (2) develop and study a procedure for creating proficiency score distributions which takes into account the errors in parameter estimation which occur during the estimation procedure.

Grantee: RAND Corporation
P. Investigator: David Grissmer
Project Title: Analyzing State NAEP Data to Identify State Educational Policies/characteristics which Cost-effectively Increase Achievement

This project will use the most recent NAEP data available to study why some states are making much larger gains in math and reading than other states, and why students with similar family characteristics score so differently across states. This work will expand the case studies of specific states previously conducted by the applicant to include six additional states selected for their particularly high or low growth rates in achievement or their high or low estimated achievement scores for students from comparable backgrounds.

Publications:

Grissmer, David W., A. Flanagan, J. Kawata and S. Williamson, Improving Student Achievement: What State NAEP Scores Tell Us,, MR-924-EDU, RAND, Santa Monica, CA, 2000

Grantee: American Institutes for Research
P. Investigator: Jon Cohen
Project Title: A Hypertext Textbook for NAEP Statistical Methods

This project proposes to develop an on-line, hypertext textbook covering existing and emerging NAEP statistical methods that can be deployed as part of the AM statistical package. The hypertext textbook will be a component of the AM software's help system. The textbook will allow users to immediately access explanations of the major components of the NAEP design and of the new estimation procedures on which the AM software is based. By helping researchers understand NAEP's underpinnings, this tool may significantly expand the NAEP user community.

Publications:

NAEPHyperTextbook, Statistical Reference for the AM Software (1999) Washington, DC: American Institutes for Research.

Grantee: Educational Testing Service
P. Investigator: Barbara Storms
Project Title: Analyzing Classroom Writing Assignments: Lessons Learned from the 1998 NAEP Classroom Writing Study

This project will use the 1998 NAEP classroom writing study data to identify the key features of classroom writing assignments which appear to lead to high-level student writing performance. The goal of this work is to develop a model of writing assignments and general classroom writing practices that will be useful to school administrators, curriculum specialists, and classroom teachers.

Publications:

Storms, B., Riazantseva, A. & Gentile, C. Designing Effective Writing Assignments. (2000). California English Journal, October, 2001. (Distributed to all elementary & secondary English teachers in California)

Eidmann-Aadahl, Elise. (2001). A National Study of Writing Assignments. National Writing Project Newsletter, November, 2001. (Distributed to over 10,000 elementary & secondary English teachers)

Gentile, C., Riazantseva, A., & Storms, B. (In Prep.). Writing Assignments: What works and Doesn't Work. Submitted to Research on the Teaching of English (revisions underway). (Main readership: university faculty doing research on writing)

Grantee: University of Southern California
P. Investigator: Eddie Ip
Project Title: Assessing the Psychometric Effects of Item Clustering Around Passages in NAEP

This project will investigate the impact of the effects of clustering items around reading passages on the estimation of NAEP proficiency scores. In addition identifying and measuring local dependency in the NAEP data, the study will develop an alternative estimation model which accounts for local dependency, and compare the ability of the new and the existing NAEP estimation models to estimate the latent structure of data which simulate the NAEP long term trend reading assessment.

Publications:

Scott, S., and Ip, E. (2002). Empirical Bayes and Item-Clustering Effects in a Latent Variable Hierarchical Model: A Case Study From the National Assessment of Educational Progress, Journal of the American Statistical Association, 97, 409-419.

Ip, E. (2002).Locally Dependent Latent Trait Model and the Dutch Identity Revisited. Psychometrika, 67, 367-386.

Grantee: University of Maryland
P. Investigator: John Guthrie
Project Title: Effects of Integrated Instruction and Reading Time on Reading Achievement in Middle School: A Policy Analysis of the NAEP Data

This study will combine data from the 1994 NAEP reading teacher questionnaire and the 1994 student reading background questions to form constructs representing integrated instruction in reading, and student engagement with reading. The study will then investigate the relationship between these constructs and student reading achievement and attempt to develop a model of optimal instructional time and student reading time that could potentially reading achievement.

Publications:

Guthrie, J.T., Schafer, W. D., & Wuang, C. W. (2001). Benefits of opportunity to read and balanced reading instruction on the NAEP. Journal of Educational Research, 94(3), 145-162.

Grantee: Educational Testing Service
P. Investigator: Eiji Muraki
Project Title: Application of Multiple-Group Generalized partial Credit Model to NAEP Linking Procedures

One major objective of the NAEP is the measurement of trends in student performance across time. To achieve this, results of different assessment cycles are linked across time periods. The methodology used to equate scores during this linking can introduce error into the estimates of student ability. This study will investigate the amount of uncertainty introduced into the NAEP ability distribution estimates by the linking strategy currently used in operational NAEP and explore several other strategies which have been proposed for operational linking.

Back to Top

Projects Funded in Fiscal Year 1998

Grantee: American Institutes for Research
P. Investigator: Donald McLaughlin
Project Title: Item-based Linked Scaling of NAEP and State Assessments

This project proposes to use and evaluate a score-based linking method known as the "variant-item technique" to calibrate state assessment items directly onto the NAEP scale. If successful, this project will develop a more precise procedure for using a state's assessments to measure achievement on the NAEP scale. This project will: (1) estimate the precision of item-based linkages, (2) assess the impact of systematic school-level variation on linkage accuracy, (3) assess the extent to which background information may be needed for a neutral scoring of state assessment items on the NAEP scale, and (4) characterize state assessment items that carry more and less weight in estimating the NAEP construct.

Grantee: Datametrics Research, Inc.
P. Investigator: Neal Thomas
Project Title: Assessing the Contribution of Background Data for Primary NAEP Reporting

Extensive background information is used in all NAEP estimation procedures, but a relatively small number of these background variables are used in primary reporting. This study proposes to estimate how much accuracy is gained (or lost) in the primary reporting of NAEP by including (or excluding) extensive background data. The study will estimate how many additional students would need to be sampled to retain the current accuracy of NAEP reports, if the background data are not used when forming primary reports. These estimates will result in a clear assessment of the costs/benefits of using background variables in primary NAEP reporting.

Publications:

Thomas, N. (2002). The role of secondary covariates when estimating latent trait population distributions. Psychometrica, 67(1), 33-48.

Grantee: CTB/McGraw-Hill
P. Investigator: Richard Patz
Project Title: Comprehensive Methodology for the Analysis of Rater Errors & their Impact on NAEP

This project proposes to develop methodology that will: (1) use the second ratings or "double reads" of NAEP open-ended items to obtain more information about students; and (2) to more appropriately quantify the error due to the rating process when open-ended items are used. The project will use a hierarchical rater model and an adaptation of an item-bundle modeling approach both of which were developed by the principal investigators. Software to implement these new procedures will be developed and publicly shared.

Publications:

Patz, R. J., Junker, B. W., Johnson, M. S., and Mariano, L. T. (2002). "The hierarchical rater model for rated test items and its application to large-scale educational assessment data," Journal of Educational and Behavioral Statistics. (In press)

Grantee: Temple University
P. Investigator: Jeremy Finn
Project Title: Taking Mathematics in High School: Is Opportunity Equal?

This project will use the 1994 NAEP Transcript study data in an attempt to examine the factors that affect mathematics course taking patterns—factors that promote course taking among some students while limiting the opportunities of others. The specific objects of the project are: (1) to identify mathematics course-taking patterns among high-school students; (2) to characterize the relationship of mathematics course-taking with family characteristics and school-related outcomes; (3) to identify inequities in course-taking according to the characteristics of students or their schools; and (4) to examine "alterable" school policies and practices that impact on students' course-taking decisions

Publications:

Finn, J., Gerber, S. and Wang, M. (2002) Course offerings, course requirements, and course taking in mathematics. Journal of Curriculum and Supervision, 17(4), 336-366.

Finn, J. (1998) Taking foreign languages in high school. Foreign Language Annals, 31(3), 287-306.

Grantee: University of Maine
P. Investigator: Jeakyung Lee
Project Title: Understanding Rural Student Achievement: Identifying Instructional and Organizational Differences between Rural and Non-rural Schools

This study proposes to examine the factors that have contributed to the recent improvement in rural student achievement and to identify the sources of the achievement gaps between rural and non-rural students. The study hypothesizes that rural schools, as compared to their non-rural counterparts, have both facilitative (e.g., small classes, supportive ethos, and safe/orderly climate) and constraining (e.g., lack of well-trained teachers and instructional resources and low availability of advanced courses) conditions which mediate the impact of their locational "disadvantage" on student achievement.

Publications:

Lee, J. (2002). Understanding Differences between Rural and Nonrural Schools Impacting Student Achievement: Implications for Title I Programs and Assessments. In K. K. Wong & M. C. Wang & (Eds.). Accountability and Efficiency in Title I Schoolwide Reform. Greenwich, CT: Information Age Publishing.

Lee, J. (2001). Interstate Variations in Rural Student Achievement and Schooling Conditions. ERIC Clearinghouse on Rural Education and Small Schools. ERIC Digest. December 2001.

Lee, J. & McIntire, W. (2001). Interstate Variation in the Mathematics Achievement of Rural and Nonrural Students. Journal of Research in Rural Education. 16 (3), 168-181.

Grantee: University of Maryland
P. Investigator: John Guthrie
Project Title: Effects of Reading Time and Instruction on Reading Achievement: A Policy Analysis of the NAEP Data

This study proposes to use the NAEP data to investigate the relationship between time spent reading and student reading achievement. Both the direct hypothesis, assumed by many standards-based school reform movements—that increased reading time relates directly to higher reading achievement—and more complex hypotheses involving the mediating affects of classroom instructional practices will be investigated.

[Results reported in publication for second grant: Guthrie, J.T., Schafer, W. D., & Wuang, C. W. (2001). Benefits of opportunity to read and balanced reading instruction on the NAEP. Journal of Educational Research, 94(3), 145-162.]

Back to Top


Last updated 28 March 2008 (NB)

Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education