Skip Navigation

Statistical Standards
Statistical Standards Program
 
Table of Contents
 
Introduction

 
·Purpose of Standards
·Background of Standards
·Development of Standards
·OMB Quality Guidelines
 

1. Development of Concepts and Methods
2. Planning and Design of Surveys
3. Collection of Data
4. Processing and Editing of Data
5. Analysis of Data / Production of Estimates or Projections
6. Establishment of Review Procedures
7. Dissemination of Data
 
Glossary
Appendix A
Appendix B
Appendix C
Appendix D
 
Publication information

For help viewing PDF files, please click here
INTRODUCTION


OMB Quality Guidelines

Background
Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554), directed the U.S. Office of Management and Budget (OMB) to issue government-wide guidelines that "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies." Information, as defined by OMB, includes any communication or representation of knowledge, such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative or audiovisual forms. Dissemination refers to any agency initiated or sponsored distribution of information to the public (OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, February 22, 2002, 67 FR 8452-8460).

NCES provides the public with a wide variety of information about the condition of American education. Information quality is important to NCES because educators, researchers, policymakers, and the public use NCES products for a variety of purposes. Thus it is important that information products that NCES disseminates are accurate and reliable. Most of the information products are available both as printed and electronic documents. They are announced on the NCES website (nces.ed.gov), and most electronic versions can be accessed and downloaded directly from the website.

Purpose and Scope
NCES guidelines have been identified as Standards for the last 15 years, thus we will retain that label. The purpose of these Standards is to describe NCES policy and procedures for reviewing and substantiating the quality of information before it is disseminated. These Standards are consistent with those issued by OMB and the Department of Education. These Standards represent a performance goal for NCES and are intended to improve the quality of the information NCES shares with the public.

In addition to the NCES Standards, the Department of Education and OMB have more general Information Quality Guidelines that apply to NCES. What is more, NCES will follow the request for corrections and appeal process described in the Department Information Quality Guidelines. www.ed.gov/policy/gen/guid/infoqualguide.html

The Standards are applicable to any information that NCES disseminates after October 1, 2002. In addition, some previously released information products continue to be used for decision-making or are relied upon by the Department of Education and the public as official, authoritative, government data; these data are, in effect, constantly being re-disseminated and thus are subject to these Standards and to the Department and OMB Information Quality Guidelines. Previously released information products that do not meet these criteria are considered archived information and thus are not subject to the Guidelines.

In addition to archived reports, these Standards do not cover all other information held or disseminated by NCES. The Department of Education Information Quality Guidelines include a list of excluded items, although that list also applies to NCES, the items that are particularly relevant to NCES are included here. For example, the guidelines generally do not cover: internal information such as employee records; internal procedural, operational, or policy manuals prepared for the management and operations of the Department of Education (and NCES) that are not primarily intended for public dissemination; information collected or developed by NCES that is not disseminated to the public, including documents intended only for inter-agency or intra-agency communications; opinions that are clearly identified as such, and that do not represent facts or NCES views; correspondence with individuals; comments received from the public in response to Federal Register notices, electronic links to information on other websites; and research findings published by NCES data cooperatives or grantees, unless NCES represents or uses the information as the official position of the Department, or in support of the official position of the Department, or has authority to review and approve the information before release.

For information covered by Information Quality Guidelines, the NCES Standards provide a basic standard of quality that can be defined based on the three elements of quality as defined by OMB: utility, objectivity, and integrity. These elements are intended to ensure that information disseminated by the NCES is useful, accurate, reliable, unbiased, and secure.

Framework
Utility refers to the usefulness of the information to its intended users. The usefulness of information disseminated by NCES should be considered from the perspective of NCES, educators, education researchers, policymakers, and the public. Utility is achieved by staying informed of information needs and developing new products and services where appropriate.

NCES wants to ensure that information it disseminates meets the needs of the intended users. NCES relies upon internal reviews and analyses, along with feedback from advisory committees, educators, education researchers, policymakers, and the public to ensure that information disseminated by NCES meets the needs of intended users. In addition, all information products should be grammatically correct and clearly written in plain English. The target audience should be clearly identified, and the product should be understandable to that audience.

Consistent with OMB guidance, the goal is to maximize the usefulness of information and minimize the cost to the government and the public. When disseminating its information products, NCES will utilize all feasible and available dissemination channels so that the public, education researchers, and policymakers can locate NCES information in an equitable and timely fashion.

The information disseminated by NCES includes administrative and statistical data. NCES collects and disseminates administrative data from universe collections of elementary and secondary and postsecondary institutions. These universe collections are based on reports aggregated from records from schools, school districts, and states. NCES also collects and disseminates data from a number of sample survey data collections that are designed to fill the information needs for statistical data. NCES supports both ongoing sample survey data collections and special purpose surveys that are designed to fill data gaps or information needs that are identified through internal review, legislative mandates, or input from data users outside the Department. All statistical reports and related products are reviewed to ensure their usefulness to the intended users. Where appropriate, contact information is available on each publication to facilitate feedback and questions by users.

The specific NCES standards that contribute directly to the utility and the dissemination of information include those on the Initial Planning of Surveys(1-1), Publication and Product Planning (1-2), and the Release and Dissemination of Reports and Data Products (7-3).

Objectivity refers to whether information is accurate, reliable, unbiased, and is presented in an accurate, clear, and unbiased manner. It involves both the content of the information and the presentation of the information. This includes complete, accurate, and easily understood documentation of the source of the information, with a description of the sources of any errors that may affect the quality of the data, when appropriate. Objectivity is achieved by using reliable information sources and appropriate techniques to prepare information products.

NCES strives to present information to the public in an accurate, clear, complete, and unbiased manner. Prior to dissemination to the public, all products are reviewed for objectivity using sound statistical methods and the principles of transparency and reproducibility, as delineated in the OMB Information Quality Guidelines. In addition, all products undergo editorial and technical peer review to assist NCES in meeting this goal.

NCES is committed to the principles for objectivity in administrative and statistical data that are outlined in the Department of Education's Guidelines. To that end, we have specific standards that relate to each of the Department's principles:

  1. In formulating a data collection plan goals of the study should be clearly described ¾ Initial Planning of Surveys (1-1), Design of Surveys (2-1), Developing a Request for Proposal (RFP) for Surveys (2-3).
     
  2. The subjects to be studied and the data to be collected should be clearly defined, using broadly understood concepts and definitions ¾ Initial Planning of Surveys (1-1), Codes and Abbreviations (1-4), Defining Race and Ethnicity Data (1-5), Design of Surveys (2-1), Developing a Request for Proposal (RFP) for Surveys (2-3), Maintaining Data Series (2-5).
     
  3. The data collection techniques should be well thought out, clearly articulated, and designed to use state of the art methodologies in the data collection ¾ Initial Planning of Surveys (1-1), Design of Surveys (2-1), Survey Response Rate Parameters (2-2), Developing a Request for Proposal (RFP) for Surveys (2-3), Pretesting Survey Systems (2-4), Educational Testing (2-6), Coverage for Frames and Samples (3-1), Achieving Acceptable Response Rates (3-2), Monitoring and Documenting Survey Contracts (3-3).
     
  4. In designing the work, every effort should be made to minimize the amount of time required for survey participants ¾ Achieving Acceptable Response Rates (3-2).
     
  5. The source of data should be reliable. In the case of sample survey data, the sample should be drawn from a complete list of items to be tested or evaluated, the appropriate respondents must be identified, correctly sampled, and queried with survey instruments that have been properly developed and tested ¾ Initial Planning of Surveys (1-1), Design of Surveys (2-1), Pretesting Survey Systems (2-4), Coverage for Frames and Samples (3-1).
     
  6. Response rates should be monitored during data collection. When necessary, appropriate steps should be taken to ensure the respondents are a representative sample ¾ Computation of Response Rates (1-3), Survey Response Rate Parameters (2-2), Achieving Acceptable Response Rates (3-2), Monitoring and Documenting Survey Contracts (3-3), Nonresponse Bias Analysis (4-4).
     
  7. Care should be taken to ensure the confidentiality of personally identifiable data, as required by law, during data collection, processing, and analysis of the resulting data ¾ Maintaining Confidentiality (4-2).
     
  8. Upon completion of the work, the data should be processed in a manner sufficient to ensure that the data are cleaned and edited to help ensure that the data are accurate and reliable ¾ Initial Planning of Surveys (1-1), Design of Surveys (2-1), Monitoring and Documenting Survey Contracts (3-3), Editing and Imputation of Item Nonresponse (4-1), Evaluation of Surveys (4-3).
     
  9. he data collection should be properly documented and stored, and the documentation should include an evaluation of the quality of the data with a description of any limitations of the data ¾ Monitoring and Documenting Survey Contracts (3-3), Documenting a Survey System (3-4), Machine Readable Products (7-1).
     
  10. Data should be capable of being reproduced or replicated based on information included in the documentation including, for example:

    a) The source(s) of the information;
    b) The date the information was current;
    c) Any known limitations on the information;
    d) The reason why the information is provided;
    e) Descriptions of any statistical techniques or mathematical operations applied to source data; and
    f) Identification of other sources of potentially corroborating or conflicting information.

    The relevant standards include ¾ Monitoring and Documenting Survey Contracts (3-3), Documenting a Survey System (3-4), Machine Readable Products (7-1), Survey Documentation in Reports (7-2).

  11. If secondary analysis of data is employed, the source should be acknowledged, the reliability of the data should be confirmed and documented, and any shortcomings or explicit errors should be acknowledged (e.g., the representativeness of the data, measurement error, data preparation error, processing error, sampling errors, and nonresponse errors) ¾ Survey Documentation in Reports (7-2).
     
  12. The analysis should be selected and implemented to ensure that the data are correctly analyzed using modern statistical techniques suitable for hypothesis testing. Techniques may vary from simple tabulations and descriptive analysis to multivariate analysis of complex interrelationships. Care should be taken to ensure that the techniques are appropriate for the data and the questions under inquiry ¾ Statistical Analysis, Inference, and Comparisons (5-1), Variance Estimation (5-2), Rounding (5-3), Tabular and Graphic Presentations of Data (5-4).
     
  13. Reports should also include the reason the information is provided, its potential uses, and cautions as to inappropriate extractions or conclusions, and the identification of other sources of corroborating or conflicting information ¾ Survey Documentation in Reports (7-2).
     
  14. Descriptions of the data and all analytical work should be reported in sufficient detail to ensure that the findings could be reproduced using the same data and methods of analysis; this includes the preservation of the data set used to produce the work ¾ Monitoring and Documenting Survey Contracts (3-3), Documenting a Survey System (3-4), Evaluation of Surveys (4-3), Machine Readable Products (7-1), Survey Documentation in Reports (7-2).
     
  15. All reports, data, and documentation should undergo editorial and technical review to ensure accuracy and clarity prior to dissemination. Qualified technical staff and peers outside the Department should do the technical review ¾ Review of Reports and Data Products (6-1).
     
  16. To ensure the utility of the work, all work must be conducted and released in a timely manner ¾ Publication and Product Planning (1-2), Release and Dissemination of Reports and Data Products (7-3).
     
  17. There should be established procedures to correct any identified errors. These procedures may include the publication of errata sheets, revised publications, or Web postings ¾ Review of Reports and Data Products (6-1), Release and Dissemination of Reports and Data Products (7-3).

Integrity refers to the security or protection of information from unauthorized access or revision. Integrity ensures that the information is not compromised through corruption or falsification.

NCES has in place appropriate security provisions for the protection of confidential information that is contained in all identified systems of records. In accordance with statutory and administrative provisions governing the protection of information, NCES protects administrative records and sample survey data that include personally identifiable information, especially survey data that are collected under a pledge of confidentiality. Applicable provisions governing the protection of information include the following:

  • Privacy Act;
  • Computer Security Act of 1987;
  • Freedom of Information Act;
  • OMB Circulars A-123, A-127, and A-130;
  • Federal Policy for the Protection of Human Subjects;
  • Government Information Security Reform Act; and
  • Education Sciences Reform Act of 2002
  • E-Government Act of 2002

The relevant standard is Maintaining Confidentiality (4-2).

Influential Information
The OMB guidelines for implementing section 515 recognize that some government information needs to meet higher quality standards than a basic standard of quality. The level of effort required to ensure the quality of information is tied to the uses of the information. Information that is defined as "influential" requires a higher level of effort to ensure its' quality and reproducibility. Scientific, financial, and statistical information is considered influential if the Department can reasonably determine that the information is likely to have a clear and substantial impact on important public policies or private sector decisions if disseminated.

Influential information must be accompanied by supporting documentation that allows an external user to clearly understand the steps involved in producing the information and, to be able to reproduce the information. Any influential original data files must describe the design, collection, and processing of the data in sufficient detail that an interested third party could understand the specifics of the original data and, if necessary, independently replicate the data collection. In the case of influential analytic results, the mathematical and statistical processes used to produce the report must be described in sufficient detail to allow an independent analyst to substantially reproduce the findings using the original data and identical methods.

When full public access to NCES data and methods is not possible due to other compelling interests, NCES will apply especially rigorous robustness checks to analytic results and will document the checks that were undertaken. In those cases where protecting the confidentiality of individually identifiable data precludes the full release of a data file, persons seeking access to such data and methods are required to follow applicable NCES requirements and procedures for seeking such access. In all cases, the interest in transparency of the agency's data shall not override other compelling interests such as privacy, intellectual property, and other confidentiality protections (16 CFR 4.9-4.11 and OMB Guidelines, par V.b.3.ii.B.j.).

Inasmuch as it is not always possible to predict in advance all of the uses of the information included in NCES data collections, all information collected and disseminated by NCES is held to the standards of quality, reproducibility, and documentation that are required for influential information.

CHART A. NCES ORGANIZATION