Search Results: (1-15 of 106 records)
|NCES 2017147||Best Practices for Determining Subgroup Size in
Accountability Systems While Protecting Personally
Identifiable Student Information
The Every Student Succeeds Act (ESSA) of 2015 (Public Law 114-95) requires each state to create a plan for its statewide accountability system. In particular, ESSA calls for state plans that include strategies for reporting education outcomes by grade for all students and for economically disadvantaged students, students from major racial and ethnic groups, students with disabilities, and English learners. In their plans, states must specify a single value for the minimum number of students needed to provide statistically sound data for all students and for each subgroup, while protecting personally identifiable information (PII) of individual students. This value is often referred to as the "minimum n-size."
Choosing a minimum n-size is complex and involves important and difficult trade-offs. For example, the selection of smaller minimum n-sizes will ensure that more students' outcomes are included in a state's accountability system, but smaller n-sizes can also increase the likelihood of the inadvertent disclosure of PII. Similarly, smaller minimum n-sizes enable more complete data to be reported, but they may also affect the reliability and statistical validity of the data.
To inform this complex decision, Congress required the Institute of Education Sciences (IES) of the U.S. Department of Education to produce and widely disseminate a report on "best practices for determining valid, reliable, and statistically significant minimum numbers of students for each of the subgroups of students" (Every Student Succeeds Act of 2015 (ESSA 2015), Public Law 114-95). Congress also directed that the report describe how such a minimum number "will not reveal personally identifiable information about students." ESSA prohibits IES from recommending any specific minimum number of students in a subgroup (Section 9209).
IES produced this report to assist states as they develop accountability systems that (1) comply with ESSA; (2) incorporate sound statistical practices and protections; and (3) meet the information needs of state accountability reporting, while still protecting the privacy of individual students.
As presented in this report, the minimum n-size refers to the lowest statistically defensible subgroup size that can be reported in a state accountability system. Before getting started, it is important to understand that the minimum n-size a state establishes and the privacy protections it implements will directly determine how much data will be publicly reported in the system.
|REL 2015077||Comparing Methodologies for Developing an Early Warning System: Classification and Regression Tree Model Versus Logistic Regression
The purpose of this report was to explicate the use of logistic regression and classification and regression tree (CART) analysis in the development of early warning systems. It was motivated by state education leaders' interest in maintaining high classification accuracy while simultaneously improving practitioner understanding of the rules by which students are identified as at-risk or not at-risk readers. Logistic regression and CART were compared using data on a sample of grades 1 and 2 Florida public school students who participated in both interim assessments and an end-of-the year summative assessment during the 2012/13 academic year. Grade-level analyses were conducted and comparisons between methods were based on traditional measures of diagnostic accuracy, including sensitivity (i.e., proportion of true positives), specificity (proportion of true negatives), positive and negative predictive power, and overall correct classification. Results indicate that CART is comparable to logistic regression, with the results of both methods yielding negative predictive power greater than the recommended standard of .90. Details of each method are provided to assist analysts interested in developing early warning systems using one of the methods.
|REL 2015071||How Methodology Decisions Affect the Variability of Schools Identified as Beating the Odds
Schools that show better academic performance than would be expected given characteristics of the school and student populations are often described as "beating the odds" (BTO). State and local education agencies often attempt to identify such schools as a means of identifying strategies or practices that might be contributing to the schools' relative success. Key decisions on how to identify BTO schools may affect whether schools make the BTO list and thereby the identification of practices used to beat the odds. The purpose of this study was to examine how a list of BTO schools might change depending on the methodological choices and selection of indicators used in the BTO identification process. This study considered whether choices of methodologies and type of indicators affect the schools that are identified as BTO. The three indicators were (1) type of performance measure used to compare schools, (2) the types of school characteristics used as controls in selecting BTO schools, and (3) the school sample configuration used to pool schools across grade levels. The study applied statistical models involving the different methodologies and indicators and documented how the lists schools identified as BTO changed based on the models. Public school and student data from one midwest state from 2007-08 through 2010-11 academic years were used to generate BTO school lists. By performing pairwise comparisons among BTO school lists and computing agreement rates among models, the project team was able to gauge the variation in BTO identification results. Results indicate that even when similar specifications were applied across statistical methods, different sets of BTO schools were identified. In addition, for each statistical method used, the lists of BTO schools identified varied with the choice of indicators. Fewer than half of the schools were identified as BTO in more than one year. The results demonstrate that different technical decisions can lead to different identification results.
|REL 2014064||Reporting What Readers Need to Know about Education Research Measures: A Guide
This brief provides five checklists to help researchers provide complete information describing (1) their study's measures; (2) data collection training and quality; (3) the study's reference population, study sample, and measurement timing; (4) evidence of the reliability and construct validity of the measures; and (5) missing data and descriptive statistics. The brief includes an example of parts of a report's methods and results section illustrating how the checklists can be used to check the completeness of reporting.
|NCES 2014097||NCES Statistical Standards
This publication contains the 2012 revised statistical standards and guidelines for the National Center for Education Statistics (NCES). These standards and guidelines are intended for use by NCES staff and contractors to guide them in their data collection, analysis, and dissemination activities. They are also intended to present a clear statement for data users regarding how data should be collected in NCES surveys, and the limits of acceptable applications and use. Users should be cognizant that the contents of this publication are continually being reviewed for technological and statistical advances.
|NCES 2013009REV||Highlights From TIMSS 2011: Mathematics and Science Achievement of U.S. Fourth- and Eighth-Grade Students in an International Context
The Trends in International Mathematics and Science Study (TIMSS) 2011 is the fifth administration of this international comparative study since 1995 when first administered. TIMSS is used to compare over time the mathematics and science knowledge and skills of fourth- and eighth-graders. TIMSS is designed to align broadly with mathematics and science curricula in the participating countries. The results, therefore, suggest the degree to which students have learned mathematics and science concepts and skills likely to have been taught in school. In 2011, there were 54 countries and 20 other educational systems that participated in TIMSS, at the fourth- or eighth-grade level, or both.
The focus of the report is on the performance of U.S. students relative to their peers in other countries in 2011, and on changes in mathematics and science achievement since 2007 and 1995. For a number of participating countries and education systems, changes in achievement can be documented over the last 16 years, from 1995 to 2011. This report also describes achievement within the United States by sex, race/ethnicity, and enrollment in public schools with different levels of poverty. In addition, it describes achievement in nine states that participated in TIMSS both as part of the U.S. national sample of public and private schools as well as individually with state-level samples of public schools.
In addition to numerical scale results, TIMSS also includes international benchmarks. The TIMSS international benchmarks provide a way to interpret the scale scores by describing the types of knowledge and skills students demonstrate at different levels along the TIMSS scale.
After the initial release of the NCES TIMSS 2011 national report and supplemental tables, several minor changes were made to the report, Appendix A, and to footnotes in Appendix E. View the errata notice for details.
|NCSER 20133000||Translating the Statistical Representation of the Effects of Education Interventions Into More Readily Interpretable Forms
This new Institute of Education Sciences (IES) report assists with the translation of effect size statistics into more readily interpretable forms for practitioners, policymakers, and researchers. This paper is directed to researchers who conduct and report education intervention studies. Its purpose is to stimulate and guide researchers to go a step beyond reporting the statistics that represent group differences. With what is often very minimal additional effort, those statistical representations can be translated into forms that allow their magnitude and practical significance to be more readily understood by those who are interested in the intervention that was evaluated.
|NCES 2012151||Statistical Methods for Protecting Personally Identifiable Information in the Disclosure of Graduation Rates of First-Time, Full-Time Degree- or Certificate-Seeking Undergraduate Students by 2-Year Degree-Granting Institutions of Higher Education
This Technical Brief provides guidance to Title IV 2-year degree-granting institutions in meeting the statutory disclosure requirement related to graduation rates while minimizing the risk of revealing the graduation status of individual students.
|NCEE 20124016||Estimating the Impacts of Educational Interventions Using State Tests or Study-Administered Tests
State assessments provide a relatively inexpensive and increasingly accessible source of data on student achievement. In the past, rigorous evaluations of educational interventions typically administered standardized tests selected by the researchers ("study-administered tests") to measure student achievement outcomes. Increasingly, researchers are turning to the lower cost option of using state assessments for measures of student achievement.
|NCES 2011609||NCES Handbook of Survey Methods
This publication presents explanations of how each survey program in NCES obtains and prepares the data it publishes. The Handbook aims to provide users of NCES data with the information necessary to evaluate the suitability of the statistics for their needs, with a focus on the methodologies for survey design, data collection, and data processing.
|NCES 2011607||National Institute of Statistical Sciences Configuration and Data Integration Technical Panel: Final Report
NCES asked the National Institute of Statistical Sciences (NISS) to convene a technical panel of survey and policy experts to examine potential strategies for configuration and data integration among successive national longitudinal education surveys. In particular the technical panel was asked to address two related issues: how could NCES configure the timing of its longitudinal studies (e.g., Early Childhood Longitudinal Study [ECLS], Education Longitudinal Study [ELS], and High School Longitudinal Study [HSLS]) in a maximally efficient and informative manner. The main, but not sole, focus was at the primary and secondary levels; and what could NCES do to support data integration for statistical and policy analyses that cross breakpoints between longitudinal studies. The NISS technical panel delivered its report to NCES in 2009. The principle recommendations included in the report are: 1. The technical panel recommended that NCES should configure K-12 studies as a series of three studies: (i) a K-5 study, followed immediately by (ii) a 6-8 study, followed immediately by (iii) a 9-12 study. One round of such studies, ignoring postsecondary follow-up to the 9-12 study, requires 13 years to complete. 2. The technical panel also recommended that budget permitting; NCES should initiate a new round of K-12 studies every 10 years. This can be done in a way that minimizes the number of years in which multiple major assessments occur. The panel found that there is no universal strategy by means of which NCES can institutionalize data integration across studies. One strategy was examined in detail: continuation of students from one study to the next. Based on experiments conducted by NISS the technical panel found that: 3. the case for continuation on the basis that it supports cross-study statistical inference is weak. Use of high-quality retrospective data that are either currently available or are likely to be available in the future can accomplish nearly as much at lower cost. 4. Continuation is problematic in at least two other senses: first, principled methods for constructing weights may not exist and, second, no matter how much NCES might advise to the contrary, researchers are likely to attempt what is likely to be invalid or uninformative inference on the basis of continuation cases alone. 5. The technical panel urged that, as an alternative means of addressing specific issues that cross studies, NCES consider the expense and benefit of small, targeted studies that target specific components of studentís trajectories.
|NCES 2011608||National Institute of Statistical Sciences Data Confidentiality Technical Panel: Final Report
NCES asked the National Institute of Statistical Sciences (NISS) to convene a technical panel of survey and policy experts to examine the NCES current and planned data dissemination strategies for confidential data with respect to: mandates and directives that NCES make data available; current and prospective technologies for protecting and accessing confidential data, as well as for breaking confidentiality; and the various user communities for NCES data and these communitiesí uses of the data. The principle goals of the technical panel were to review the NCES current and planned data dissemination strategies for confidential data, assessing whether these strategies are appropriate in terms of both disclosure risk and data utility, and then to recommend to NCES any changes that the task force deems desirable or necessary. The NISS technical panel delivered its report to NCES in 2008. The report included four principal recommendations, the first three of which confirmed existing NCES strategies and practices:
|NCES 2011603||Statistical Methods for Protecting Personally Identifiable Information in Aggregate Reporting
This Statewide Longitudinal Data Systems (SLDS) Technical Brief examines what protecting student privacy means in a reporting context. To protect a studentís privacy, the studentís personally identifiable information must be protected from public release. When schools, districts, or states publish reports on studentsí educational progress, they typically release aggregated dataódata for groups of studentsóto prevent disclosure of information about an individual. However, even with aggregation, unintended disclosures of personally identifiable information may occur. Current reporting practices are described and each is accompanied by an example table that is used to consider whether the intended protections are successful.
The Brief also illustrates that some practices work better than others in protecting against disclosures of personally identifiable information about individual students. Each data protection practice requires some loss of information. The challenge rests in identifying practices that protect information about individual students, while at the same time minimizing the negative impact on the utility of the publicly reported data. Drawing upon the review and analysis of current practices, the Brief concludes with a set of recommended reporting rules that can be applied in reports of percentages and rates that are used to describe student outcomes to the public. These reporting rules are intended to maximize the amount of detail that can be safely reported without allowing disclosures from student outcome measures that are based on small numbers of students.
NCES welcomes comments on the recommended reporting rules.
|NCSER 20103006||Statistical Power Analysis in Education Research
This paper provides a guide to calculating statistical power for the complex multilevel designs that are used in most field studies in education research. For multilevel evaluation studies in the field of education, it is important to account for the impact of clustering on the standard errors of estimates of treatment effects. Using ideas from survey research, the paper explains how sample design induces random variation in the quantities observed in a randomized experiment, and how this random variation relates to statistical power. The manner in which statistical power depends upon the values of intraclass correlations, sample sizes at the various levels, the standardized average treatment effect (effect size), the multiple correlation between covariates and the outcome at different levels, and the heterogeneity of treatment effects across sampling units is illustrated. Both hierarchical and randomized block designs are considered. The paper demonstrates that statistical power in complex designs involving clustered sampling can be computed simply from standard power tables using the idea of operational effect sizes: effect sizes multiplied by a design effect that depends on features of the complex experimental design. These concepts are applied to provide methods for computing power for each of the research designs most frequently used in education research.
|NCES 201002||2008/09 Baccalaureate and Beyond Longitudinal Study (B&B:08/09) Field Test Methodology Report - Working Paper Series
This report describes the methodology and findings for the field test of the 2008/2009 Baccalaureate and Beyond Longitudinal Study (B&B:08/09). These students, who completed their bachelorís degree requirements during the 2007-08 academic year, were first interviewed as part of the 2008 National Postsecondary Student Aid Study (NPSAS:08) field test. BPS:08/09 is the first follow-up of this cohort. The B&B:08/09 field test was used to plan, implement, and evaluate methodological procedures, instruments, and systems proposed for use in the full-scale study scheduled for the year following graduation from a bachelorís degree program. The report provides the sampling design and methodologies used in the field test. It also describes data collection outcomes, including response rates, interview burden, and results of incentive, mailing, and prompting experiments. In addition, the report provides details on the evaluation of data quality for reliability of responses, item nonresponse, and question delivery and data entry error. Recommendations for the full-scale study are provided for the sampling design, locating and tracing procedures, interviewer training, data collection, and instrumentation.