Skip Navigation

Statistical Standards
Statistical Standards Program
 
Table of Contents
 
Introduction
1. Development of Concepts and Methods
2. Planning and Design of Surveys
3. Collection of Data

 
3-1 Coverage for Frames and Samples
3-2 Achieving Acceptable Response Rates
3-3 Monitoring and Documenting Survey Contracts
3-4 Documenting a Survey System

4. Processing and Editing of Data
5. Analysis of Data / Production of Estimates or Projections
6. Establishment of Review Procedures
7. Dissemination of Data
 
Glossary
Appendix A
Appendix B
Appendix C
Appendix D
 
Publication information

For help viewing PDF files, please click here
COLLECTION OF DATA

SUBJECT: DOCUMENTING A SURVEY SYSTEM

NCES STANDARD: 3-4

PURPOSE: To ensure that complete documentation is kept on NCES surveys and survey systems and their associated contract deliverables. Documentation includes those materials necessary to understand how to properly analyze data from each survey, as well as the information necessary to replicate and evaluate each survey. In addition, survey system documentation includes information necessary to design and estimate resource requirements of future similar surveys.

KEY TERMS: coverage, edit, frame, imputation, instrument, nonsampling error, public-use data file, response rate, sampling error, strata, survey system, and variance.


STANDARD 3-4-1. Survey system documentation must include all information necessary to properly analyze the data. This information shall, at a minimum, include the following:

  1. Final data set(s);
     
  2. Final instrument(s) or a facsimile thereof;
     
  3. Definitions of all variables;
     
  4. Data file layout;
     
  5. Descriptions of constructed variables on the data file that are computed from responses to other variables on the file;
     
  6. Description of variables used to uniquely identify cases in the data file;
     
  7. Description of sample weights and how to apply them;
     
  8. Description of the strata and primary sampling unit (PSU) identifiers to be used for analysis;
     
  9. Description of how to calculate variances appropriate for the survey design;
     
  10. Description of all imputation methods applied to the data and how to remove imputed values from the data; and
     
  11. Descriptions of known data anomalies and corrective actions.

    GUIDELINE 3-4-1A: If the data are collected through a Web-based collection or through a CATI or CAPI interview, the following information should be included in the documentation of the final instruments:

    1. All items in the instrument (e.g., questions, check items, and help screens);
       
    2. Items extracted from other data files to pre-fill the instrument (e.g., dependent data from a prior round of interviewing); and
       
    3. Items that are input to the post data collection processing steps (e.g., output of an automated instrument).
       

STANDARD 3-4-2: To insure that a survey can be replicated and properly evaluated, the survey system documentation must, at a minimum, include the following:

  1. Justifications for the items on the survey instrument, including how the final items were selected;
     
  2. All instructions to respondents and/or interviewers either about how to properly respond to a survey item or how to properly present a survey item;
     
  3. Description of the data collection methodology;
     
  4. Sampling plan and justifications for why it was implemented, and, if possible, the final sample frame;
     
  5. Selected sample;
     
  6. Description of the magnitude of sampling error associated with the survey, and how it was calculated;
     
  7. Description of the sources of nonsampling error associated with the survey (e.g., coverage, measurement);
     
  8. Unit response rates (weighted and unweighted);
     
  9. Overall response rates (weighted and unweighted);
     
  10. Item response rates; and
     
  11. Total response rates.

    GUIDELINE 3-4-2A: The survey system documentation should also include the following:

    1. Final weighting plan specifications, including calculations for how the final weights were derived, and justifications for why it was implemented;
       
    2. Final imputation plan specifications and justifications for why it was implemented;
       
    3. Data editing plan specifications and justifications for why it was implemented; and
       
    4. Data processing plan specifications and justifications for why it was implemented;

    GUIDELINE 3-4-2B: Where appropriate, methods for bounding or estimating the nonsampling error from each source identified in the evaluation plan should be developed and implemented.

    GUIDELINE 3-4-2C: Where possible, nonsampling error estimates and bounds should make use of data from other surveys or from administrative records or censuses, taking into account the limitations of the external data.

    GUIDELINE 3-4-2D: For recurring surveys, a quality profile report that itemizes all sources of identified error should be produced. Where possible, estimates or bounds on the magnitudes of these errors should be provided; the total error model for the survey should be discussed; and the survey should be assessed in terms of this model.


STANDARD 3-4-3: To insure that NCES has sufficient information to design future surveys and to accurately estimate their resource requirements, survey system documentation must include the following:

  1. All information germane to the contractual operation of the survey, including the request for proposals used to solicit the contract(s);
     
  2. Independent government cost estimate;
     
  3. Contract(s) used to develop, conduct, and report on the survey;
     
  4. Any modifications to the contract(s);
     
  5. Final contract deliverables, progress reports, and vouchers; and
     
  6. Office of Management and Budget (OMB) clearance package and correspondence with OMB about survey clearance.
     

STANDARD 3-4-4: At a minimum, survey documentation must be stored electronically in a format that can be viewed without proprietary software. Final data sets shall be stored in ASCII format. Additional copies in other formats are allowed, but ASCII versions are required. In addition, substantive reports written to release the data shall also be stored, at a minimum, in the format originally used to produce the report, and PDF or ASCII (see Standard 7-1).


STANDARD 3-4-5: All reports, documentation, and public-use data file must be stored on the Web, a CD-ROM, or an NCES dedicated server. Restricted data files and associated documentation must be transmitted to the Statistical Standards Program for secure storage.