Distance Education at Postsecondary Education Institutions: 1997-98

Survey Methodology and Data Reliability

Postsecondary Education Quick Information System

The Postsecondary Education Quick Information System (PEQIS) was established in 1991 by the National Center for Education Statistics, U.S. Department of Education. PEQIS is designed to conduct brief surveys of postsecondary institutions or state higher education agencies on postsecondary education topics of national importance. Surveys are generally limited to two or three pages of questions, with a response burden of about 30 minutes per respondent. Most PEQIS institutional surveys use a previously recruited, nationally representative panel of institutions. The PEQIS panel was originally selected and recruited in 1991–92. In 1996, the PEQIS panel was reselected to reflect changes in the postsecondary education universe that had occurred since the original panel was selected. A modified Keyfitz approach was used to maximize overlap between the 1996 panel and the 1991–92 panel. The sampling frame for the PEQIS panel recruited in 1996 was constructed from the 1995– 96 Integrated Postsecondary Education Data System (IPEDS) "Institutional Characteristics" file. Institutions eligible for the PEQIS frame for the panel recruited in 1996 included 2-year and 4- year (including graduate-level) institutions (both institutions of higher education and other postsecondary institutions), and less-than-2-year institutions of higher education located in the 50 states and the District of Columbia: a total of 5,353 institutions.

The PEQIS sampling frame for the panel recruited in 1996 was stratified by instructional level (4-year, 2-year, less-than-2-year), control (public, private nonprofit, private for-profit), highest level of offering (doctor's/firstprofessional, master's, bachelor's, less than bachelor's), total enrollment, and status as either an institution of higher education or other postsecondary institution. Within each of the strata, institutions were sorted by region (Northeast, Southeast, Central, West), whether the institution had a relatively high minority enrollment, and whether the institution had research expenditures exceeding $1 million. The sample of 1,669 institutions was allocated to the strata in proportion to the aggregate square root of total enrollment. Institutions within a stratum were sampled with equal probabilities of selection. The modified Keyfitz approach resulted in 80 percent of the institutions in the 1996 panel overlapping with the 1991–92 panel. Panel recruitment was conducted with the 338 institutions that were not part of the overlap sample. During panel recruitment, 20 institutions were found to be ineligible for PEQIS, primarily because they had closed or offered just correspondence courses. The final unweighted response rate at the end of PEQIS panel recruitment with the institutions that were not part of the overlap sample was 98 percent (312 of the 318 eligible institutions). The final participation rate across the 1,669 institutions that were selected for the 1996 panel was 1,628 participating institutions out of 1,634 eligible institutions. There were 1,634 eligible institutions because 15 institutions in the overlap sample were determined to be ineligible for various reasons.

Each institution in the PEQIS panel was asked to identify a campus representative to serve as survey coordinator. The campus representative facilitates data collection by identifying the appropriate respondent for each survey and forwarding the questionnaire to that person.

Sample and Response Rates

The sample for this survey consisted of all of the institutions in the PEQIS panel, for a sample of 1,612 institutions.²⁷ In October 1998, questionnaires (see appendix C) were mailed to the PEQIS coordinators at the institutions. Coordinators were told that the survey was designed to be completed by the person(s) at the institution most knowledgeable about the institution's distance education course offerings. Eleven institutions were found to be out of the scope of the survey because they were closed, leaving 1,601 eligible institutions. These 1,601 institutions represent the universe of approximately 5,010 2-year and 4-year (including graduate-level) postsecondary education institutions in the 50 states and the District of Columbia. Telephone followup of nonrespondents was initiated in November 1998; data collection and clarification was completed in March 1999. For the eligible institutions that received surveys, an unweighted response rate of 93 percent (1,487 responding institutions divided by the 1,601 eligible institutions in the sample) was obtained. The weighted response rate for this survey was also 93 percent. The unweighted overall response rate was 93 percent (99.6 percent panel recruitment participation rate multiplied by the 92.9 percent survey response rate). The weighted overall response rate was 92 percent (99.7 percent weighted panel recruitment participation rate multiplied by the 92.7 percent weighted survey response rate).

Weighted item nonresponse rates ranged from 0 percent to 2.6 percent. Item nonresponse rates for most items were less than 1 percent. The items with the highest nonresponse rates involved the numbers of courses and enrollments. Because one of the major reasons for conducting this survey was to make national estimates for these numbers, imputation was implemented for all item nonresponse in the survey. The imputation procedures involved a combination of hot-deck imputation procedures for questions 2, 3, and 4, involving numbers of courses and enrollments, and the assignment of modal values from imputation classes for question 10, concerning plans for distance education technologies. These procedures are described in more detail below. There were 33 institutions, which represent about 4.4 percent of the 753 responding institutions offering a distance education course, that did not report one or more data items in questions 2, 3, and 4. However, only a very few institutions did not report the numbers of different college-level, credit-granting courses offered by general fields of study in question 4. To impute the missing numbers of courses and enrollments, hot-deck imputation procedures were used. Hot-deck imputation selects a donor value from another institution with similar characteristics to use as the imputed value. The institutions were grouped to form imputation classes according to sector (public, private nonprofit, private for-profit), institutional level (4-year, 2-year) and enrollment size class. A donor institution was selected among the responding institutions within the imputation class. The donor institution was selected randomly, if the total number of credit granting courses and total enrollment were not reported. However, if these totals were reported, the institution with the closest totals to the recipient institution was selected as the donor. If enrollment was not reported but the number of courses offered was reported, enrollment for a specific field of study was imputed by multiplying the reported number of courses offered by the institution with the ratio of the total enrollment to the total number of courses offered for the field of study, computed using the responding institutions within the imputation class.

There were 29 institutions (or about 3 percent of the total number of institutions eligible for question 10) with missing data in one or more of the parts of question 10. The imputed items for question 10 had to be consistent with the corresponding reported data items in question 9. The institutions were grouped into imputation classes according to sector, institutional level, enrollment size class, and the reported value for question 9. The reported modal value among the responding institutions within the imputation class was assigned as the imputed value for the missing data item in question 10.

Sampling and Nonsampling Errors

The response data were weighted to produce national estimates (see table A). The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability. The survey estimates are also subject to nonsampling errors that can arise because of nonobservation (nonresponse or noncoverage) errors, errors of reporting, and errors made in data collection. These errors can sometimes bias the data. Nonsampling errors may include such problems as misrecording of responses; incorrect editing, coding, and data entry; differences related to the particular time the survey was conducted; or errors in data preparation. While general sampling theory can be used in part to determine how to estimate the sampling variability of a statistic, nonsampling errors are not easy to measure and, for measurement purposes, usually require that an experiment be conducted as part of the data collection procedures or that data external to the study be used.

To minimize the potential for nonsampling errors, the questionnaire was pretested with respondents at institutions like those that completed the survey. During the design of the survey and the survey pretest, an effort was made to check for consistency of interpretation of questions and to eliminate ambiguous items. The questionnaire and instructions were extensively reviewed by the National Center for Education Statistics, U.S. Department of Education. Manual and machine editing of the questionnaire responses were conducted to check the data for accuracy and consistency. Cases with missing or inconsistent items were recontacted by telephone. Data were keyed with 100 percent verification.

Variances

The standard error is a measure of the variability of estimates due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of institutions reporting that they offered distance education courses in 1997–98 is 33.6 percent, and the estimated standard error is 1.0 percent. The 95 percent confidence interval for the statistic extends from [33.6 - (1.0 times 1.96)] to [33.6 + (1.0 times 1.96)], or from 31.6 to 35.6 percent. Tables of standard errors for each table and figure in the report are provided in appendix B.

Estimates of standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The mean squared error of the replicate estimates around the full sample estimate provides an estimate of the variances of the statistics.²⁸ To construct the replications, 51 stratified subsamples of the full sample were created and then dropped one at a time to define 51 jackknife replicates.²⁹ A computer program (WesVarPC), distributed free of charge by Westat through the Internet,³⁰ was used to calculate the estimates of standard errors. WesVarPC is a stand-alone Windows application that computes sampling errors for a wide variety of statistics (totals, percents, ratios, log-odds ratios, general functions of estimates in tables, linear regression parameters, and logistic regression parameters). The test statistics used in the analysis were calculated using the jackknife variances and thus appropriately reflected the complex nature of the sample design. In particular, an adjusted chisquare test using Satterthwaite's approximation to the design effect was used in the analysis of the two-way tables.³¹ Finally, Bonferroni adjustments were made to control for multiple comparisons where appropriate. For example, for an "experiment-wise" comparison involving g pairwise comparisons, each difference was tested at the 0.05/g significance level to control for the fact that g differences were simultaneously tested.

Definitions of Analysis Variables

The following institutional characteristics were used as variables for analyzing the survey data:

Type of institution: public 2-year, private 2- year, public 4-year, private 4-year. Type was created from a combination of level (2-year, 4-year) and control (public, private). Twoyear institutions are defined as institutions at which the highest level of offering is at least 2 but less than 4 years (below the baccalaureate degree); 4-year institutions are those at which the highest level of offering is 4 or more years (baccalaureate or higher degree).³² Private comprises private nonprofit and private for-profit institutions; these private institutions are reported together because there are too few private for-profit institutions in the sample for this survey to report them as a separate category. Postsecondary education institutions include both institutions of higher education (traditional colleges and universities) and other postsecondary institutions (e.g., allied health and vocational-technical schools). Less-than-2-year institutions are not included in the PEQIS panel or in this survey.
Size of institution: less than 3,000 students (small); 3,000 to 9,999 students (medium); and 10,000 or more students (large).

Comparing the PEQIS Studies: Technical Notes and Limitations

There are a number of factors that must be considered when comparing the 1995 and 1997– 98 PEQIS studies. Differences in the samples and variations in question wording are discussed here. In addition, the data from the 1995 study were not imputed for item nonresponse. However, item nonresponse was very low and did not substantially affect the results. Item nonresponse is noted where it occurred when the data are presented in chapter 7.

Differences in the Samples

The two studies were sent to two somewhat different groups of institutions. The sample for the 1995 study consisted of all of the 2-year and 4-year higher education institutions in the PEQIS panel selected in 1991–92, which was based on the 1990–91 IPEDS "Institutional Characteristics" file. This sample of 1,274 institutions (of which 1,203 were respondents) represented the universe of approximately 3,460 higher education institutions at the 2-year and 4- year level in the 50 states, the District of Columbia, and Puerto Rico estimated to exist at the time of the survey. At the time the 1991–92 PEQIS panel was selected, NCES defined higher education institutions as institutions that are accredited at the college level by an agency recognized by the Secretary of the U.S. Department of Education.³³ Higher education institutions are a subset of all postsecondary education institutions. The 1995 study was sent only to higher education institutions in the PEQIS panel.

The sample for the 1997–98 study consisted of all of the 2-year and 4-year postsecondary education institutions in the PEQIS panel selected in 1996, which was based on the 1995–96 IPEDS "Institutional Characteristics" file. The 1996 PEQIS panel was selected in a way that maximized the overlap between the 1991–92 and 1996 panels. The 1,601 institutions in the 1997– 98 study represented the universe of approximately 5,010 postsecondary education institutions at the 2-year and 4-year level in the 50 states and the District of Columbia estimated to exist at the time of the survey. At the time the 1996 PEQIS panel was selected, NCES was still defining higher education institutions in the same way as it was when the 1991–92 PEQIS panel was selected. The 1997–98 study was sent to all postsecondary education institutions in the PEQIS panel, both higher education and other postsecondary education institutions. In order to make comparisons between the two studies, the data from the 1997–98 study were analyzed for the subset of higher education institutions. These 1,347 institutions (of which 1,244 were respondents) represented the universe of approximately 3,580 higher education institutions at the 2-year and 4-year level in the 50 states and the District of Columbia estimated to exist at the time of the study. It is the data from this subset of institutions that are presented in the chapter on changes in distance education since 1994–95.

Variations in Question Wording

Number of distance education courses offered. In the 1995 study, institutions were asked for the total number of distance education courses with different catalog numbers offered by the institution in academic year 1994–95. Courses with different catalog numbers excluded multiple sections of the same course. In the 1997–98 study, institutions were asked to report the total number of different distance education courses (including courses for all levels and audiences), and the number of different college-level, creditgranting distance education courses offered at the institution in the 12-month 1997–98 academic year. If a course had multiple sections or was offered multiple times during the year, institutions were instructed to count it as only one course.

Enrollment in distance education courses. In 1995, institutions that offered any distance education courses were asked how many students were formally enrolled in the institution's distance education courses in academic year 1994–95. In the 1997–98 study, institutions that offered any distance education courses were asked about the total enrollment in all distance education courses in the 12-month 1997–98 academic year (including enrollments in courses designed for all levels and types of students, including elementary/secondary, adult education, etc.), and the enrollment in college-level, creditgranting distance education courses in 1997–98.

In both studies, if a student was enrolled in multiple courses, institutions were instructed to count the student for each course in which he or she was enrolled. Thus, enrollments may include duplicated counts of students.

Degree and certificate programs. In 1995, institutions were asked whether students could complete degrees or certificates by taking distance education courses exclusively, and if so, how many different degrees or certificates could be received in this way. In the 1997–98 study, institutions were asked if they had any collegelevel degree or certificate programs based on credit-granting courses designed to be completed totally through distance education, and if so, how many such degree and certificate programs they had. In 1997–98, they were also instructed to include programs that may require a small amount of on-campus coursework or labwork, clinical work in hospitals, or similar arrangements, and to include baccalaureate degree completion programs.

Distance education technologies. In 1995, institutions were asked which types of technology they used to deliver their distance education courses. In 1997–98, institutions were asked which types of technology they used as a primary mode of instructional delivery for distance education courses. An individual course could only have one predominant mode of delivery. Institutions could, however, indicate that they used many different technologies as primary modes of instructional delivery across all of their distance education courses, since different distance education courses could use different types of technology. Information was not collected in either year about the number of courses offered using each technology, only whether the institution used it at all (or used it as a primary mode of instruction) for distance education courses.

The lists of technologies in the 2 years were similar but not identical. The incidence of audiographics was so low in 1995 that it was not included as a separate category in 1997–98. CDROM and multi-mode packages were added as categories in 1997–98. The wording of the computer-based technologies was changed to more accurately reflect how these technologies are used. In 1995, the categories were two-way online (computer-based) interactions during instruction, and other computer-based technology (e.g., Internet). In 1997–98, the categories were Internet courses using synchronous (i.e., simultaneous) computer-based instruction (e.g., interactive computer conferencing or Interactive Relay Chat), and Internet courses using asynchronous (i.e., not simultaneous) computerbased instruction (e.g., e-mail, listserves, and most World Wide Web-based courses). For the comparisons presented in this report, two-way online (computer-based) interactions during instruction is compared to Internet courses using synchronous computer-based instruction, and other computer-based technology (e.g., Internet) is compared to Internet courses using asynchronous computer-based instruction.

Background Information

The survey was performed under contract with Westat, using the Postsecondary Education Quick Information System (PEQIS). This is the ninth PEQIS survey to be conducted. Westat's Project Director was Elizabeth Farris, and the Survey Manager was Laurie Lewis. Bernie Greene was the NCES Project Officer.

The following individuals reviewed this report:

Outside NCES

Gary Adams, California Community Colleges
Stephanie Cronen, American Institutes for Research
Ann Hiros, Consortium of Distance Education
David Hurst, Education Statistics Services Institute
James Lubell, Office of Postsecondary Education, U.S. Department of Education
Becky Smerdon, American Institutes for Research
Karen Spahn, University of Phoenix

Inside NCES

Ellen Bradburn, Early Childhood, International, and Crosscutting Studies Division
Chris Chapman, Early Childhood, International, and Crosscutting Studies Division
Paula Knepper, Postsecondary Studies Division
Roslyn Korb, Postsecondary Studies Division
Marilyn McMillen, Chief Statistician

For more information about the Postsecondary Education Quick Information System or the Survey on Distance Education at Postsecondary Education Institutions, contact:

Bernie Greene
Early Childhood, International, and Crosscutting Studies Division
National Center for Education Statistics Office of Educational Research and Improvement
U.S. Department of Education
555 New Jersey Avenue, NW
Washington, DC 20208-5651
Email: Bernard_Greene@ed.gov
Telephone: (202) 219-1366

²⁷ The number of institutions in the PEQIS panel decreased from 1,628 to 1,612 because of institutional closures and mergers.

²⁸ K. Wolter. Introduction to Variance Estimation, Springer-Verlag, 1985.

²⁹ Ibid, 183.

³⁰ WesVarPC version 2 is available through the Internet at http://www.westat.com/wesvar /.

³¹ For example, see J.N.K. Rao and A. Scott, "On Chi-square Tests for Multi-way Contingency Tables with Cell Proportions Estimated from Survey Data," Annals of Statistics 12 (1984): 46-60

³² Definitions for level are from the data file documentation for the Integrated Postsecondary Education Data System (IPEDS) "Institutional Characteristics" file, U.S. Department of Education, National Center for Education Statistics.

³³ NCES now defines higher education institutions as those institutions that are eligible for Title IV financial aid programs and that grant degrees (i.e., awarded at least one associate's or higher degree in the previous academic year). In 1997–98, there were 4,096 higher education institutions that met this revised definition, out of a universe of 9,632 postsecondary education institutions (U.S. Department of Education, National Center for Education Statistics, Postsecondary Institutions in the United States: 1997– 98. Washington, DC: 1999).

Top