Survey Methodology and Reliability

The population of interest for this survey was institutions of higher education (IHEs) that serve freshmen and are accredited at the college-level by an association or agency recognized by the Secretary of Education. A national probability sample of 546 IHEs was selected from a universe of 3,283 colleges and universities. The sampling frame used for the survey was the universe file of the Higher Education General Information System (HEGIS) Fall Enrollment and Compliance Report of Institutions of Higher Education of 1983-84. Of the total initial sample of 546 institutions, 47 were determined to be out of scope, mainly because they did not have freshmen. The weighted total from the 473 responding institutions in the sample (out of the 499 eligible institutions) is 2,874, representing all colleges and universities with freshmen (Table 17). The weighted total from the institutions able to report remedial figures was somewhat lower (Table 18) (see discussion of item non response rates below).

Questionnaires (copy included) were mailed in late April 1990. The questionnaire and cover letter addressed to an experienced survey coordinator at the institution requested that the questionnaire be completed by the person at the institution most knowledgeable about remedial/ developmental studies. Data collection and follow up efforts continued through mid-July. An overall response rate of 95 percent was obtained from the eligible institutions.

The universe was stratified by type of control, type of institution, and enrollment size. Within strata, schools were selected at uniform rates but the sampling rates varied considerably from stratum to stratum. The response data were weighted to produce national estimates and a weight adjustment was made to account for survey non response. The weights were calculated for each institution inversely proportional to its square root of size. These weights ranged from 1.9636 to 24.2000. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability. If the questionnaire had been sent to a different sample, the responses would not have been identical; some figures might have been higher, while others might have been lower.

The standard error is a measure of the variability due to sampling when estimating statistics. It indicates the variability in the population of possible estimates of a parameter for a given sample size. Standard errors can be used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of freshmen enrolled in remedial mathematics courses at public institutions in fall 1989 is 21 percent, and the estimated standard error is 1.0. The 95 percent confidence internal for the statistic extends from 21- (1.0 times 1.96) to 21 + (1.0 times 1.96), or from 19 to 23 percent. This means that one can be confident that this interval contains the true population parameter 95 percent of the time.

Estimates of standard errors were computed using a replication technique known as jackknife replication. The estimated standard errors for some key statistics are shown in Table 19, Table 19 con't 2, Table 19 con't 3, Table 19 con't 4. In some cases, estimates of standard errors were relatively large because statistics were based on a small number of cases. This was true, for example, for schools designated as minority status (those with a student body less than 50 percent white). Standard errors for statistics not included in this table can be obtained from NCES upon request.

For categorical data, relationships between variables with 2 or more levels have been tested using chi-square tests at the .05 level of significance, adjusted for average design effect. If the overall chi-square test was significant, it was followed up with pair-wise tests using a Bonferroni t statistic, which maintained an overall 95 percent confidence level or better.

Survey estimates are also subject to errors of reporting and errors made in the collection of the data. These non sampling errors can sometimes bias the data. while general sampling theory can be used to determine how to estimate the sampling variability of a statistic, non sampling errors are not easy to measure and usually require that an experiment be conducted as part of the data collection procedures or the use of data external to the study.

Non sampling errors may include such problems as differences in the respondents' interpretation of the meaning of the questions, differences related to the particular time the survey was conducted, or errors in data preparation. During the design of the survey and survey pretest, an effort was made to check for consistency of interpretation of questions and to eliminate ambiguous items. The questionnaire was pretested with respondents like those who completed the survey, and the questionnaire and instructions were extensively reviewed by the National Center for Education Statistics (NCES) and a panel of specialists in remedial/developmental studies. Manual and machine editing of the questionnaires was conducted to check the data for accuracy and consistency. Cases with missing or inconsistent items were recontacted by telephone; data were keyed with 100 percent verification.

Data are presented for all institutions and by the following institutional characteristics: type (2-year and 4-year), control (public and private), geographic region (Northeast, Central, Southeast, and West), enrollment size (less than 1,000 undergraduates, 1,000 to 4,999 undergraduates, and 5,000 or more undergraduates), minority status (less than 50 percent white, and greater than or equal to 50 percent white). Some data on the percentage of institutions offering remedial courses are also presented by selectivity ratings (most difficult, very difficult, moderately difficult, minimally difficult, and noncompetitive).

Region classifications are those used by the Bureau of Economic Analysis of the U.S. Department of Commerce, the National Assessment of Educational Progress, and the National Education Association. The Northeast includes Connecticut, Delaware, the District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York Pennsylvania, Rhode Island, and Vermont. The Central region includes Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin. The Southeast includes Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia. The West includes Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming.

Item non response rates varied. Non response rates for items discussed in the "Characteristics of Remedial Courses and Programs" (pages 7-15) ranged from 0.0 percent to 0.6 percent. Non response rates for items on the number of teachers of remedial courses were slightly higher, ranging from 3.9 percent to 7.2 percent. As mentioned previously, the non response rates for freshman enrollment and passing items were considerably higher, as some institutions were unable to provide these figures and were reluctant to give estimates. Therefore, imputations were made for the following missing freshman enrollment and passing rates:

Items requiring imputations	Number of cases imputed
Percent enrolled in remedial reading courses	55
Percent enrolled in remedial writing courses	61
Percent enrolled in remedial mathematics courses	68
Percent passing remedial reading courses	73
Percent passing remedial writing courses	79
Percent passing remedial mathematics courses	88
Percent enrolled in remedial courses in reading writing or mathematics	78

Imputations for the first six items were done initially. Of the 473 responding institutions, 361 offered at least one remedial program. Of these 361 schools, item imputations rates for the six items ranged from 15.2 percent to 24.4 percent.

The 94 schools requiring imputation were first broken into three classes: 52 schools needed all six variables imputed; 14 needed all three passing rates imputed, but none of the enrollment rates; and 28 needed some other combination of variables imputed. In order to minimize the impact of imputation on both averages and variances, a hot-deck imputation procedure was used, respecting the sampling stratification wherever possible. Hot-deck imputation selects a donor value from another institution with similar characteristics to use as the imputed value. Thus, the institutions were sorted by strata and within strata by total school size before beginning imputation.

Imputations were then done for the 66 schools that needed imputation for all three passing rates (and possibly all three enrollment rates). A single donor institution was selected for all missing data for a given institution, if it was the institution immediately preceding the one needing imputation, and if it contained values for all six variables. Minimizing the number of times a single institution is used as a donor minimizes the impact on variance. Therefore, if an institution had already been used as a donor, the preceding eligible institution on the list was used. If all three of the preceding potential donors had already been used, a donor institution would be used a second time. This kept the donor institution as similar in size to the imputed institution as possible.

For 12 of the remaining 28 cases needing imputation, some of the enrollment (and/or passing) data were reported. For these cases, the missing data were imputed from the other data reported by the same institution. For example, if the institution reported that 30 percent of its students were enrolled in remedial reading classes and 40 percent enrolled in remedial mathematics, but did not report the percent for writing, the average, 35 percent, was imputed for remedial writing.

This left 16 institutions needing imputation for one or two enrollment (and/or passing ) variables where no data were reported for the other subjects. (In addition, one institution had one missing and one reported enrollment variable and two missing passing variables. The enrollment imputation followed the procedure outlined in the previous paragraph, and the passing variables were imputed as described in this paragraph. Thus, 17 rather than 16 schools were in this category.) These were imputed using the same hot-deck procedure described earlier.

As a result of the above procedures, three institutions were each used as donors three times and seven other institutions were each used twice.

The imputed values had a small and statistically insignificant impact on the estimated overall average percentage of students enrolled in or passing remedial classes. Comparing the pre-imputation averages with those after imputation shows that including imputed values raised the percentage enrolled by 1.4 percent for reading and writing, and 2.2 percent for mathematics. It lowered the passing rates by 0.4 percent for reading and 0.2 percent for mathematics, while raising the rate by 0.4 percent for writing.

Imputations for the last item--total percentage of freshmen enrolled in one or more remedial courses in reading, writing, or mathematics--were restricted by the values for the percentage enrolled in each of the individual subjects (remedial reading writing, and mathematics). The minimum value for the total unduplicated percentage enrolled in remedial courses equals the largest percentage enrolled in remedial reading, writing, or mathematics. The maximum value for the total, unduplicated percentage enrolled in remedial courses equals the sum of the percentages enrolled in remedial reading, writing, or mathematics. Because of these restrictions, it was decided to impute the midpoint between the minimum and maximum values.

The imputed values for this item had a slightly larger but still statistically insignificant impact on the estimated overall average percentage of students enrolled in one or more remedial courses. Including imputed values raised the percentage enrolled by 4.7 percent. The appropriateness of using the midpoint as the value to be imputed was confirmed by examining those cases where no values were imputed for percentages enrolled in individual remedial subjects or for the total, unduplicated percentage. For institutions without any imputations for these items, the value of the total, unduplicated percentage enrolled was 43 percent of the difference between the minimum value and the maximum value.

The survey was performed under contract with Westat, Inc., using the Fast Response Survey System (FRSS). Westat's Project Director was Elizabeth Farris, and the Survey Manager was Wendy Mansfield. Jeffrey Williams was the NCES Project Officer through data collection and follow up efforts. Judi Carpenter was the NCES Project Officer during the remainder of the survey (through analysis and report writing). The data requester was MacKnight Black, Education Program Officer, Postsecondary Education Statistics Division. FRSS was designed to collect quickly, and with minimal burden on respondents, small quantities of data needed for educational planning and policy.

Top