Appendix A: Technical Notes

Fast Response Survey System

The Fast Response Survey System (FRSS) was established in 1975 by the National Center for Education Statistics (NCES), U.S. Department of Education. FRSS is designed to collect issue-oriented data within a relatively short timeframe. FRSS collects data from state education agencies, local education agencies, public and private elementary and secondary schools, public school teachers, and public libraries. To ensure minimal burden on respondents, the surveys are generally limited to three pages of questions, with a response burden of about 30 minutes per respondent. Sample sizes are relatively small (usually about 1,000 to 1,500 respondents per survey) so that data collection can be completed quickly. Data are weighted to produce national estimates of the sampled education sector. The sample size permits limited breakouts by classification variables. However, as the number of categories within the classification variables increases, the sample size within categories decreases, which results in larger sampling errors for the breakouts by classification variables.

Sample Design

The sample for the FRSS survey on distance education courses consisted of 2,305 public school districts in the 50 states and the District of Columbia. It was selected from the 2001–02 NCES Common Core of Data (CCD) Local Education Agency Universe file, which was the most current file available at the time of selection. The sampling frame included 14,229 regular public school districts and 989 "other education agencies" with at least one charter school. For the purposes of the study, "regular" school districts included any local school district that was not a component of a supervisory union (i.e., Education Agency type 1 on the CCD), or was a local school district component of a supervisory union sharing a superintendent and administrative services with other local school districts (i.e., Education Agency type 2 on the CCD). Excluded from the sampling frame were districts in the outlying U.S. territories and regular districts with no enrollments.

The school district sampling frame was stratified by district type (regular or charter), enrollment size (less than 1,000, 1,000 to 2,499, 2,500 to 9,999, 10,000 to 99,999, and 100,000 or more), and percentage of children in the district ages 5-17 in families living below the poverty level (less than 10 percent, 10 to 19.99 percent, 20 to 29.99 percent, and 30 percent or more).¹ Districts in the frame were then sorted by type of locale (urban, suburban, rural) and region (Northeast, Southeast, Central, West) to induce additional implicit stratification. These variables are defined in more detail in the Definitions of Analysis Variables section of this report.

Data Collection and Response Rates

Questionnaires and cover letters for the study were mailed to the superintendent of each sampled district in November 2003. The letter introduced the study and requested that the questionnaire be completed by the district's director of curriculum and instruction, the technology coordinator, the distance education coordinator, or another staff member who was most knowledgeable about the district's distance education courses. Respondents were offered the option of completing the survey via the Web or by mail. Telephone follow-up for survey nonresponse and data clarification was initiated in December 2003 and completed at the end of April 2004.

To calculate response rates, NCES uses standard formulas established by the American Association of Public Opinion Research.² Thus, unit response rates (RRU) are calculated as the ratio of the weighted number of completed interviews (I) to the weighted number of in-scope sample cases. There are a number of different categories of cases that make up the total number of in-scope cases, including

I = weighted number of completed interviews;

R = weighted number of refused interview cases;

O = weighted number of eligible sample units not responding for reasons other than refusal;

NC = weighted number of noncontacted sample units known to be eligible;

U = weighted number of sample units of unknown eligibility, with no interview; and

e = estimated proportion of sample units of unknown eligibility that are eligible.

The unit response rate represents a composite of the components as follows:

Of the 2,305 districts in the sample, 10 were found to be ineligible for the survey because they no longer existed. Another three were found to be ineligible because they did not meet some other criteria for inclusion in the sample (e.g., the district was composed of only one school, which was a charter school that offered only prekindergarten classes, and thus was ineligible for the sample). This left a total of 2,292 eligible districts in the sample. Completed questionnaires were received from 2,158 districts, or 94 percent of the eligible districts³ (table A-1). The weighted response rate was 96 percent. The weighted number of eligible districts in the survey represent the estimated universe of public school districts in the 50 states and the District of Columbia. The estimated number of districts in the survey universe decreased from the 15,218 districts in the sampling frame to an estimated 15,040 because some of the districts were determined to be ineligible for the FRSS survey during data collection.

Imputation for Item Nonresponse

Although item nonresponse was very low, data were imputed for all missing questionnaire data. These 29 items are listed in table A-2. The missing items included both numerical data, such as counts of enrollments in distance education courses, and categorical data, such as which technologies were used as primary modes of instructional delivery for distance education courses. The missing data were imputed using a "hot-deck" approach to obtain a "donor" district from which the imputed values were derived. Under the hot-deck approach, a donor district that matched selected characteristics of the district with missing data (the recipient district) was identified. The matching characteristics included district type, region, metropolitan status, district enrollment size class, and poverty concentration. Once a donor was found, it was used to derive the imputed values for the district with missing data. For categorical items, the imputed value was simply the corresponding value from the donor district. For numerical items, the imputed value was calculated by taking the donor's response for that item (e.g., number of distance education course enrollments) and dividing that number by the total number of students enrolled in the donor district. This ratio was then multiplied by the total number of students enrolled in the recipient district to provide an imputed value. All missing items for a given district were imputed from the same donor whenever possible.

Data Reliability

While the "Distance Education Courses for Public Elementary and Secondary School Students" survey was designed to account for sampling error and to minimize nonsampling error, estimates produced from the data collected are subject to both types of error. Sampling error occurs because the data are collected from a sample rather than a census of the population, and nonsampling errors are errors made during the collection and processing of the data.

Sampling Errors

The responses were weighted to produce national estimates (see table A-1). The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability. General sampling theory was used to estimate the sampling variability of the estimates and to test for statistically significant differences between estimates.

The standard error is a measure of the variability of an estimate due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of public school districts with students regularly enrolled in distance education courses is 36.4 percent and the standard error is 1.2 percent (see tables 1 and 1-A). The 95 percent confidence interval for the statistic extends from [36.4-(1.2 x 1.96)] to [36.4 + (1.2 x 1.96)], or from 34.0 to 38.8 percent. The 1.96 is the critical value for a statistical test at the 0.05 significance level (where 0.05 indicates the 5 percent of all possible samples that would be outside the range of the confidence interval).

Because the data from the FRSS distance education courses survey were collected using a complex sampling design, the variances of the estimates from this survey (e.g., estimates of proportions) are typically different from what would be expected from data collected with a simple random sample. Not taking the complex sample design into account can lead to an underestimation of the standard errors associated with such estimates. To generate accurate standard errors for the estimates in this report, standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The mean square error of the replicate estimates around the full sample estimate provides an estimate of the variance of the statistic. To construct the replications, 50 stratified subsamples of the full sample were created and then dropped 1 at a time to define 50 jackknife replicates. A computer program (WesVar) was used to calculate the estimates of standard errors. WesVar is a stand-alone Windows application that computes sampling errors from complex samples for a wide variety of statistics (totals, percents, ratios, log-odds ratios, general functions of estimates in tables, linear regression parameters, and logistic regression parameters).

Where appropriate, estimates with a coefficient of variation (CV) greater than 50 percent have been noted. The CV is a ratio of the standard error to the estimate, multiplied by 100 to obtain a percent. The CV is used to compare the variability of two or more estimates, where higher CV values indicate greater variability and lower CV values indicate less variability.

Nonsampling Errors

Nonsampling error is the term used to describe variations in the estimates that may be caused by population coverage limitations and data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems like unit and item nonresponse,⁴ differences in respondents' interpretations of the meaning of questions, response differences related to the particular time the survey was conducted, and mistakes made during data preparation. It is difficult to identify and estimate either the amount of nonsampling error or the bias caused by this error. To minimize the potential for nonsampling error, this study used a variety of procedures, including a pretest of the questionnaire with directors of curriculum and instruction or other people at the district who were deemed to be the most knowledgeable about the district's distance education courses. The pretest provided the opportunity to check for consistency of interpretation of questions and definitions and to eliminate ambiguous items. The questionnaire and instructions were also extensively reviewed by NCES and the data requester at the Office of Educational Technology. In addition, manual and machine editing of the questionnaire responses was conducted to check the data for accuracy and consistency. Cases with missing or inconsistent items were recontacted by telephone to resolve problems. Data were keyed with 100 percent verification for surveys received by mail, fax, or telephone.

Definitions of Analysis Variables

District Enrollment Size-This variable indicates the total number of students enrolled in the district based on data from the 2001–02 CCD. Data on this variable were missing for three districts; districts with missing data were excluded from all analyses involving district enrollment size. The variable was collapsed into the following three categories:

Less than 2,500 students
2,500 to 9,999 students
10,000 or more students

Metropolitan Status-This variable indicates the type of community in which the district is located, as defined in the 2001–02 CCD (which uses definitions based on U.S. Census Bureau classifications). Metropolitan status is the classification of an education agency's service area relative to a Metropolitan Statistical Area (MSA). An MSA is an area consisting of one or more contiguous counties (cities and towns in New England) that contain a core area with a large population nucleus, as well as adjacent communities having a high degree of economic and social integration with that core. An area is defined as an MSA if it is the only MSA in the immediate area and has a city of at least 50,000 population or it is an urbanized area of at least 50,000 with a total metropolitan population of at least 100,000 (75,000 in New England). The categories are described in more detail below.

Urban-Primarily serves a central city of an MSA
Suburban-Serves an MSA but not primarily its central city
Rural-Does not serve an MSA

Region-This variable classifies districts into one of the four geographic regions used by the Bureau of Economic Analysis of the U.S. Department of Commerce, the National Assessment of Educational Progress, and the National Education Association. Data were obtained from the 2001–02 CCD Local Education Agency Universe file. The geographic regions are

Northeast-Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont

Southeast-Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia

Central-Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin

West-Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming

Poverty Concentration-This variable indicates the percentage of children in the district ages 5-17 in families living below the poverty level, based on the Title I data provided to the U.S. Department of Education by the Bureau of the Census. Data on this variable were missing for 112 districts; districts with missing data were excluded from all analyses involving poverty concentration. The variable was collapsed into the following three categories:

Less than 10 percent
10 to 19 percent
20 percent or more

Contact Information

For more information about the survey, contact Bernard Greene, Early Childhood, International, and Crosscutting Studies Division, National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, 1990 K Street NW, Washington, DC 20006; e-mail: Bernard.Greene@ed.gov; telephone (202) 502-7348.

---

¹Poverty estimates for school districts were based on Title I data provided to the U.S. Department of Education by the Bureau of the Census and contained in U.S. Department of Commerce, Bureau of the Census, Current Population Survey (CPS) "Small Area Income and Poverty Estimates, Title I Eligibility Database, 1999." The No Child Left Behind Act of 2001 directs the Department of Education to distribute Title I basic and concentration grants directly to school districts on the basis of the most recent estimates of children in poverty. For income year 1999, estimates were derived for districts according to their 2001–02 boundaries based on 2000 census data and model-based estimates of poverty for all counties. For detailed information on the methodology used to create these estimates, please refer to www.census.gov/hhes/www/saipe.html.

²See American Association for Public Opinion Research (AAPOR), Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (Ann Arbor, MI: AAPOR, 2000). Note that for this report, there were no sampled units with unknown eligibility.

³Approximately 40 percent of surveys were completed via mail, 29 percent via the Web, 13 percent via phone, and 12 percent via fax.

⁴Unit nonresponse typically refers to situations in which the survey was not completed by the respondent. Item nonresponse occurs when an item on the survey is blank or incomplete.

Top