Data Reliability

Estimates produced using data from the NHES are subject to two types of errors: sampling and nonsampling errors. Nonsampling errors are errors made in the collection and processing of data. Sampling errors occur because the data are collected from a sample, rather than a census, of the population.

Nonsampling Errors

Nonsampling error is the term used to describe variations in the estimates that may be caused by population coverage limitations and data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems like unit and item nonresponse, the differences in respondents' interpretations of the meaning of survey questions, response differences related to the particular time the survey was conducted, the tendency for respondents to give socially desirable responses, and mistakes in data preparation.

In general, it is difficult to identify and estimate either the amount of nonsampling error or the bias caused by this error. For each NHES survey, efforts were made to prevent such errors from occurring and to compensate for them where possible. For instance, during the survey design phase, cognitive interviews were conducted for the purpose of assessing respondent knowledge of the topics, comprehension of questions and terms, and the sensitivity of items. The design phase also entailed extensive staff testing of the CATI instrument and a pretest in which several hundred interviews were conducted to identify problems with the initial questionnaire.

An important nonsampling error for a telephone survey is the failure to include persons who do not live in households with telephones. Weighting adjustments using characteristics related to telephone coverage were used to reduce the bias in the estimates associated with not including students who do not live in households with telephones. From January to June 2007, the percentage of children with no telephone service was 1.7 percent, and the percentage of children with wireless (cell-phone) service only was 11.9 percent (Blumberg and Luke 2007).

Sampling Errors

The sample of households with telephones selected for each NHES survey is just one of many possible samples that could have been selected from all households with telephones. Therefore, estimates produced from each NHES survey may differ from estimates that would have been produced from other samples. This type of variability is called sampling error because it arises from using a sample of households with telephones rather than all households with telephones.

The standard error is a measure of the variability due to sampling when estimating a statistic; standard errors for estimates presented in this report were computed using a jackknife replication method. Standard errors can be used as a measure of the precision expected from a particular sample. The probability that a complete census count would differ from the sample estimate by less than 1 standard error is about 68 percent. The chance that the difference would be less than 1.65 standard errors is about 90 percent; and that the difference would be less than 1.96 standard errors, about 95 percent.

Standard errors for all of the estimates are presented in the tables. These standard errors can be used to produce confidence intervals. For example, an estimated 73 percent of students were reported to have attended an assigned public school in 2007 (table 1). This figure has an estimated standard error of 0.7. Therefore, the estimated 95 percent confidence interval for this statistic is approximately 72 to 74 percent [73 percent +/– (1.96*0.7)]. That is, in 95 out of 100 samples from the same population, the percentage of students enrolled in assigned public schools should fall between 72 and 74 percent.