Survey Methodology and Data Reliability

Postsecondary Education Quick Information System

The Postsecondary Education Quick Information System (PEQIS) was established in 1991 by the National Center for Education Statistics, U.S. Department of Education. PEQIS is designed to conduct brief surveys of postsecondary institutions or state higher education agencies on postsecondary education topics of national importance. Surveys are generally limited to two or three pages of questions, with a response burden of about 30 minutes per respondent. Most PEQIS institutional surveys use a previously recruited, nationally representative panel of institutions. The sampling frame for the PEQIS panel recruited in 1992 was constructed from the 1990-91 Integrated Postsecondary Education Data System (IPEDS) Institutional Characteristics file. Institutions eligible for the PEQIS frame for the panel recruited in 1992 included 2-year and 4-year (including graduate-level) institutions (both institutions of higher education and other postsecondary institutions), and less-than-2-year institutions of higher education located in the 50 states, the District of Columbia, and Puerto Rico: a total of 5,317 institutions.

The PEQIS sampling frame for the panel recruited in 1992 was stratified by instructional level (4-year, 2-year, less-than-2-year), control (public, private nonprofit, private for-profit), highest level of offering (doctor's/first professional, master's, bachelor's, less than bachelor's), total enrollment, and status as either an institution of higher education or other postsecondary institution. Within each of the strata, institutions were sorted by region (Northeast, Southeast, Central, West), whether the institution had a relatively high minority enrollment, and whether the institution had research expenditures exceeding $1 million. The sample of 1,665 institutions was allocated to the strata in proportion to the aggregate square root of full-timeequivalent enrollment. Institutions within a stratum were sampled with equal probabilities of selection. During panel recruitment, 50 institutions were found to be ineligible for PEQIS, primarily because they had closed or offered just correspondence courses. The final unweighted response rate at the end of PEQIS panel recruitment in spring 1992 was 98 percent (1,576 of the 1,615 eligible institutions). The weighted response rate for panel recruitment was 96 percent.

Each institution in the PEQIS panel was asked to identify a campus representative to serve as survey coordinator. The campus representative facilitates data collection by identifying the appropriate respondent for each survey and forwarding the questionnaire to that person.

Sample and Response Rates

The sample for this survey consisted of all of the 2-year and 4-year (including graduate-level) higher education institutions in the PEQIS panel, for a sample of 1,276 institutions. In late September 1995, questionnaires (see appendix B) were mailed to the PEQIS coordinators at the institutions. Coordinators were told that the survey was designed to be completed by the person(s) at the institution most knowledgeable about the institution's distance education courses.

Two institutions were found to be out of the scope of the survey because they were closed, leaving 1,274 eligible institutions. These 1,274 institutions represent the universe of approximately 3,460 2-year and 4-year (including graduate-level) higher education institutions in the 50 states, the District of Columbia, and Puerto Rico. Telephone followup of nonrespondents was initiated in late October 1995; data collection and clarification was completed in late January 1996. For the eligible institutions that received surveys, an unweighted response rate of 94 percent (1,203 responding institutions divided by the 1,274 eligible institutions in the sample) was obtained. The weighted response rate for this survey was 96 percent. The unweighted overall response rate was 92 percent (97.6 percent panel recruitment participation rate multiplied by the 94.4 percent survey response rate). The weighted overall response rate was 92 percent (96.1 percent weighted panel recruitment participation rate multiplied by the 95.6 percent weighted survey response rate).

Weighted item nonresponse rates ranged from 0 percent to 3.1 percent. Item nonresponse rates for most items were less than 1 percent. Because the item nonresponse rates were so low, imputation for item nonresponse was not implemented.

Sampling and Nonsampling Errors

The response data were weighted to produce national estimates (see table 25). The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability.

The survey estimates are also subject to nonsampling errors that can arise because of nonobservation (nonresponse or noncoverage) errors, errors of reporting, and errors made in data collection. These errors can sometimes bias the data. Nonsampling errors may include such problems as misrecording of responses; incorrect editing, coding, and data entry; differences related to the particular time the survey was conducted; or errors in data preparation. While general sampling theory can be used in part to determine how to estimate the sampling variability of a statistic, nonsampling errors are not easy to measure and, for measurement purposes, usually require that an experiment be conducted as part of the data collection procedures or that data external to the study be used.

To minimize the potential for nonsampling errors, the questionnaire was pretested with respondents at institutions like those that completed the survey. During the design of the survey and the survey pretest, an effort was made to check for consistency of interpretation of questions and to eliminate ambiguous items. The questionnaire and instructions were extensively reviewed by the National Center for Education Statistics and the National Institute on Postsecondary Education, Libraries, and Lifelong Learning, U.S. Department of Education. Manual and machine editing of the questionnaire responses were conducted to check the data for accuracy and consistency. Cases with missing or inconsistent items were recontacted by telephone. Data were keyed with 100 percent verification.

Variances

The standard error is a measure of the variability of estimates due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of institutions reporting that the institution plans to offer distance education courses in the future is 25.3 percent, and the estimated standard error is 1.6 percent. The 95 percent confidence interval for the statistic extends from [25.3 - (1.6 times 1.96)] to [25.3 + (1.6 times 1.96)], or from 22.2 to 28.4 percent. Tables of standard errors for each table and figure in the report are provided in appendix A.

Estimates of standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The mean square error of the replicate estimates around the full sample estimate provides an estimate of the variances of the statistics.7 To construct the replications, 51 stratified subsamples of the full sample were created and then dropped one at a time to define 51 jackknife replicates.8 A computer program (WesVarPC), distributed free of charge by Westat through the Internet,9 was used to calculate the estimates of standard errors. WesVarPC is a stand-alone Windows application that computes sampling errors for a wide variety of statistics (totals, percents, ratios, log-odds ratios, general functions of estimates in tables, linear regression parameters, and logistic regression parameters).

The test statistics used in the analysis were calculated using the jackknife variances and thus appropriately reflected the complex nature of the sample design. In particular, an adjusted chi-square test using Satterthwaite's approximation to the design effect was used in the analysis of the two-way tables.10 Finally, Bonferroni adjustments were made to control for multiple comparisons where appropriate. For example, for an "experiment-wise" comparison involving g pairwise comparisons, each difference was tested at the 0.05/g significance level to control for the fact that g differences were simultaneously tested.

Background Information

The survey was performed under contract with Westat, using the Postsecondary Education Quick Information System (PEQIS). This is the fifth PEQIS survey to be conducted. Westat's Project Director was Elizabeth Farris, and the Survey Managers were Debbie Alexander and Laurie Lewis. Bernie Greene was the NCES Project Officer. The data were requested by the National Institute on Postsecondary Education, Libraries, and Lifelong Learning, U.S. Department of Education.

This report was reviewed by the following individuals:

Outside NCES

• Robert Albrecht, Chief Academic Officer, Western Governors University
• Cheryl Garnette, Office of Reform Assistance and Dissemination, U.S. Department of Education
• Gregory Henschel, National Institute on Postsecondary Education, Libraries, and Lifelong Learning, U.S. Department of Education

Inside NCES

• Michael Cohen, Statistical Standards and Services Group
• Mary Frase, Data Development and Longitudinal Studies Group
• Andrew Malizio, Data Development and Longitudinal Studies Group
• Roslyn Korb, Surveys and Cooperative Systems Group
• Jeffrey Williams, Surveys and Cooperative Systems Group

For more information about the Postsecondary Education Quick Information System or the Survey on Distance Education Courses Offered by Higher Education Institutions, contact Bernie Greene, Data Development and Longitudinal Studies Group, National Center for Education Statistics, Office of Educational Research and Improvement, 555 New Jersey Avenue, NW, Washington, DC 20208-5651, e-mail: peqis@ed.gov

Top