Skip Navigation
Adult Education Participation in 2004-05

NCES 2006-077
May 2006

Appendix A: Data Reliability and Validity

Estimates produced using data from the survey are subject to two types of error, sampling and nonsampling errors. Nonsampling errors are errors made in the collection and processing of data. Sampling errors occur because the data are collected from a sample rather than the whole population.

Nonsampling Errors

Nonsampling error is the term used to describe variations in the estimates that may be caused by population coverage limitations and data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems like unit2 and item nonresponse, the differences in respondents' interpretations of the meaning of the questions, response differences related to the particular time the survey was conducted, and mistakes in data preparation. In the AE-NHES:2005 survey, efforts were made to minimize nonsampling error through cognitive testing in the survey design stage, a two-stage field test of the survey, on-line data edits and post-interview edits, and a comparison of the survey estimates with similar estimates from previous surveys.

An important source of nonsampling error for a telephone survey is the failure to include persons who do not live in households with telephones (a population coverage limitation). This is particularly problematic in RDD surveys because so little is known about the sampled telephone numbers of these individuals. The March 2005 Current Population Survey (CPS) shows that 93.3 percent of all adults ages 16 and older live in households with telephones (based on independent tabulations of the March 2005 Current Population Survey - U.S. Census Bureau 2005). Estimation procedures were used to help reduce the bias in the estimates associated with excluding the 7 percent of adults who do not live in households with telephones. An issue that has arisen in recent years is households that have cell phones rather than landlines. As more data on cell phone-only households is collected, its impact on nonsampling errors will be further addressed.

A study was conducted by Montaquila, Brick, and Brock (1997) examining telephone coverage bias for subsamples of the population in NHES:1996. This study found that with very few exceptions, the adjusted weights yielded estimates with absolute telephone coverage bias of 2 percent or less. Undercoverage bias for some subgroups may have been large due to larger proportions of persons in these subgroups residing in non-telephone households.

Another potential source of nonsampling error is respondent bias. Respondent bias occurs when respondents systematically misreport (intentionally or unintentionally) information in a study. There are many different forms of respondent bias. One of the best known is social desirability bias, which occurs when respondents give what they believe is the socially desirable response (Demaio 1984). For example, surveys that ask about whether respondents voted in the most recent election typically obtain a higher estimate of the number of people who voted than do voting records. Although respondent bias may affect the accuracy of the results, it does not necessarily invalidate other results from a survey. If there are no systematic differences among specific groups under study in their tendency to give socially desirable responses, then comparisons of the different groups will accurately reflect differences among the groups. For the AE-NHES:2005 survey, given the nature of the questions being asked, i.e., mostly informative and not related to opinions or attitudes, it is not likely that there was much social desirability bias.


Response Rates

In the AE-NHES:2005 survey, Screener interviews were completed with 58,140 households, with a weighted Screener unit response rate of 66.9 percent. A screener was used to collect information on household composition and interview eligibility. A total of 8,904 adults completed the AE interview, for a weighted unit response rate of 71.2 percent and an overall estimated unit response rate (the product of the Screener unit response rate and the AE unit response rate) of 47.6 percent.

A unit nonresponse bias analysis was undertaken for NHES:2003. This involved the examination of unit response rates as a whole and for various subgroups. The analysis was done to determine characteristics that are associated with Screener unit nonresponse, and to compare estimates of interest based on adjusted and unadjusted weights. These investigations revealed no evidence of unit nonresponse bias. An extensive unit nonresponse bias was conducted in 2001 to analyze the effect of weighting on estimates, as well as to examine the effect of various data collection procedures (refusal conversion, second refusal conversion, and varying numbers of call attempts) on the estimates (See Brick et al. forthcoming). For each hypothetical data collection scenario considered in this study, the sample was reweighted, and the estimates were compared across scenarios. For this analysis of unit nonresponse bias, as well, there was no evidence of bias in the weighted estimates as the data collection effort was varied. While such an analysis is unable to directly examine bias due to the exclusion of cases that did not respond under any of the scenarios studied, other approaches have been used in NHES to evaluate that bias, including comparisons of NHES estimates to those from other data sources. All such studies are limited in the variables that can be included; unit nonresponse bias may still be present in other variables that were not studied.

Item nonresponse (i.e., the failure to complete some items in an otherwise completed interview) was very low for most items in the AE-NHES:2005 survey. The item nonresponse rate for most variables included in this report was 3 percent or lower. The one item with nonresponse rates larger than 10 percent was the item related to household income. Items with missing data were imputed using a hot-deck procedure (Rao and Shao, 1992) in which cells are formed that contain cases with similar characteristics and a donor value is used to impute the missing value. The estimates included in this report are based on the imputed data.


Sampling Errors

The sample of telephone households selected for the AE-NHES:2005 survey is just one of many possible samples that could have been selected. Therefore, estimates produced from this sample may differ from estimates that would have been produced from other samples. This type of variability is called sampling error because it arises from using a sample of households with telephones, rather than having surveyed all households with telephones.

The standard error is a measure of the variability due to sampling when estimating a statistic; standard errors for estimates presented in this report were computed using a jackknife replication method. Standard errors can be used as a measure of the precision expected from a particular sample. The probability that a sample estimate would differ from the population parameter obtained from a complete census count by less than 1 standard error is about 68 percent. The chance that the difference would be less than 1.65 standard errors is about 90 percent; and that the difference would be less than 1.96 standard errors, about 95 percent.

Standard errors for all of the estimates are presented in the tables. These standard errors can be used to produce confidence intervals. For example, an estimated 44 percent of adults reported in 2005 that they participated in some type of formal adult education in the 12 months prior to the interview. This figure has an estimated standard error of 0.7 percent. Therefore, the estimated 95 percent confidence interval for this statistic is approximately 43 to 45 percent (44 1.96 (0.7)). That is, if the processes of selecting a sample, collecting the data, and constructing the confidence interval were repeated, it would be expected that in 95 out of 100 samples from the same population, the confidence interval would contain the true participation rate.



All of the estimates in this report are based on weighting the observations using the probabilities of selection of the respondents and other adjustments to partially account for nonresponse and coverage bias. Weights were developed to produce unbiased and consistent estimates of national totals. The weight used in this report is FAWT, the weight variable used to estimate the characteristics of adults. In addition to properly weighting the responses, special procedures for estimating the statistical significance of the estimates were employed because the NHES:2005 data were collected using a complex sample design. Complex sample designs result in data that violate some of the assumptions that are normally made when assessing the statistical significance of results from a simple random sample. Frequently, the standard errors of the estimates from these surveys are larger than would be expected if the sample was a simple random sample and the observations were independent and identically distributed random variables. Eighty replicate weights, FAWT1 to FAWT80, were used to produce estimates of the sampling errors of estimates. The estimates and standard errors presented in this report were produced using WesVar Complex Samples software and a jackknife replication procedure (Westat 2000).


2 In the AE-NHES:2005 survey, the unit nonresponse rate for the adult education component is defined as the percentage of eligible sampled adults who do not complete the survey.