Skip Navigation

Statistical Standards
Statistical Standards Program
 
Table of Contents
 
Introduction
1. Development of Concepts and Methods
2. Planning and Design of Surveys
3. Collection of Data
4. Processing and Editing of Data

 
4-1 Data Editing and Imputation of Item Nonresponse
4-2 Maintaining Confidentiality
4-3 Evaluation of Surveys
4-4 Nonresponse Bias Analysis

5. Analysis of Data / Production of Estimates or Projections
6. Establishment of Review Procedures
7. Dissemination of Data
 
Glossary
Appendix A
Appendix B
Appendix C
Appendix D
 
Publication information

For help viewing PDF files, please click here
PROCESSING AND EDITING OF DATA

SUBJECT: NONRESPONSE BIAS ANALYSIS

NCES STANDARD: 4-4

PURPOSE: To identify the existence of potential bias due to unit and item nonresponse.

KEY TERMS: base weight, frame, item nonresponse, nonresponse bias, overall unit nonresponse, potential magnitude of nonresponse bias, required response items, response rate, stage of data collection, total nonresponse, unit nonresponse, and wave.


STANDARD 4-4-1: Any survey stage of data collection with a unit or item response rate less than 85 percent must be evaluated for the potential magnitude of nonresponse bias before the data or any analysis using the data may be released. (See Standard 1-3 for how to calculate unit and item response rates.) Estimates of survey characteristics for nonrespondents and respondents are required to assess the potential nonresponse bias. The level of effort required is guided by the magnitude of the nonresponse.


STANDARD 4-4-2: When unit nonresponse is high, nonresponse bias analysis must be conducted at the unit level to determine whether or not the data are missing at random and to assess the potential magnitude of unit nonresponse bias. At the unit level, the nonresponse bias analysis must be conducted using base weights for the survey stage with nonresponse. The following guidelines must be considered in such analysis.

    GUIDELINE 4-4-2A: Comparisons of respondents and nonrespondents across subgroups using available sample frame characteristics provide information about the presence of nonresponse bias. This approach is limited because observed frame characteristics are often unrelated or weakly related to more substantive items in the survey.

    GUIDELINE 4-4-2B: Formal multivariate modeling can be used to compare the proportional distribution of characteristics of respondents and nonrespondents to determine if nonresponse bias exists and, if so, to estimate the magnitude of the bias. These multivariate analyses are used to identify the characteristics of cases least likely to respond to an interview (such analyses are often referred to as nonresponse propensity models). Cases are coded as either responding to or not responding to the interviews and multivariate techniques are used to identify which case characteristics significantly relate to unit nonresponse. The predictor variables should have very high response rates. This approach may be limited by the extent to which such predictors exist in the data.

    GUIDELINE 4-4-2C: Comparisons of respondents to known population characteristics from external sources can provide information about how the respondents differ from a known population. This approach is limited by information available from existing sources on the population of interest. Known population characteristics are often unrelated or weakly related to more substantive items in the survey.

    GUIDELINE 4-4-2D: For collections in which successive levels of effort (e.g., increasing number of contact attempts, increasing incentives to respond) are employed to reduce nonresponse, comparisons of characteristics can be made between the later/more difficult cases and the earlier/easier cases to estimate the characteristics of the remaining nonrespondents. This approach may be less effective if overall or total response rates are relatively low or if a collection period is relatively short in duration. In addition, the assumption that nonrespondents are like those respondents who are difficult to reach may not hold.

    GUIDELINE 4-4-2E: More intensive methods and/or incentives can be used to conduct a followup survey of nonrespondents on a reduced set of required response items. Comparisons between the nonrespondent followup survey and the original survey can be made to measure the potential magnitude of nonresponse bias in the original survey. This approach may be costly and less useful for modeling nonresponse bias if the nonrespondent followup survey response rates are also below 70 percent.

    GUIDELINE 4-4-2F: The estimated bias can be summarized using the following measures. One measure is the ratio of the bias to the standard error, using the base weight. A second measure is the ratio of the bias to the reported survey mean, using the base weight. If weighting adjustments are used to reduce bias, these measures should also be reported using the final weighted estimates.


STANDARD 4-4-3: When item nonresponse is high, nonresponse bias analysis must be conducted at the item level to determine whether or not the data are missing at random and to assess the potential magnitude of item nonresponse. To analyze potential bias from item nonresponse, the guidelines below must be considered.

    GUIDELINE 4-4-3A: For an item with a low total response rate, respondents and nonrespondents can be compared on sampling frame and/or questionnaire variables for which data on respondents and nonrespondents are available. Base weights must be used in such analysis. Comparison items should have very high response rates. This approach may be limited to the extent that items available for respondents and nonrespondents may not be related to the low response rate item being analyzed.

    GUIDELINE 4-4-3B: Formal multivariate modeling can be used to compare characteristics of respondents and nonrespondents to determine if nonresponse bias exists and, if so, to estimate the magnitude of the bias. These multivariate analyses are used to identify the characteristics of cases least likely to respond to an item (such analyses are often referred to as nonresponse propensity models). Cases are coded as either responding to or not responding to the item and multivariate techniques are used to identify which case characteristics significantly relate to item nonresponse. Base weights must be used in such analysis. The predictor variables should have very high response rates. This approach may be limited by the extent to which such predictors exist in the data.

    GUIDELINE 4-4-3C: If the overall response rate is acceptable, nonresponse bias analysis may be conducted using data from survey respondents only. Unit level respondents who answered the low response rate item can be compared to unit level respondents who did not answer the item. Final weights and unimputed variables should be used in such an analysis. The comparison items should have very high item response rates. This approach may be limited because it does not directly analyze nonresponse bias that may originate because of unit level nonresponse.