Skip Navigation
small header image
E.D. TAB: Curricular Differentiation in Public High Schools
NCES: 95360
December 1994

Appendix A—Survey Methodology and Data Reliability

Sample Schools Selection

The sampling frame for FRSS Survey of High School Curricular Options was the 1990-91 and Staffing Survey (SASS). The 1990-91 SASS was conducted by the National Center for Education Statistics (NCES) to collect nationally representative data on critical aspects of teaching supply and demand, the composition of the administrator and teaching work force, and the status of teaching and schooling generally. The sampling frame for the SASS was constructed from the 1988-89 NCES Common Core of Data (CCD) public school universe file, and included over 83,000 public elementary and secondary schools. Over 9,000 schools were selected from the public school frame for the 1990-91 SASS. Of these, about 4,000 are elementary schools and therefore were not eligible for this FRSS survey. Also, for this survey, secondary schools were defined as regular public schools that provide instruction to grades 10-12. Thus, the "sampling frame" for this survey was approximately 5,000 public secondary or combined schools in the 1990-91 SASS.

A stratified sample of 1,000 public schools was selected from the approximately 5,000 eligible schools in the 1990-91 SASS sample. The SASS sample was originally stratified by state and instructional level, and schools within each state/level stratum were selected with probabilities proportionate to the square root of the number of teachers in the school. For this FRSS survey, schools were stratified by geographic region, locale (Johnson 1989), and enrollment size. Within these primary strata, schools were further sorted by percentage of minority enrollment. The SASS schools were subsampled within strata at rates designed to yield a PPS (probability -proportionate-to-size) sample, where the size measure is the square root of the enrollment of the school. That is, conditional on the SASS sample, schools were selected with probabilities proportionate to the square-root of enrollment times the final SASS weight. Use of the square root of enrollment as the sampling measure of size was efficient for estimating school characteristics and quantitative measures correlated with enrollment. The allocation of the sample to the major strata was made in a manner that was expected to be reasonably efficient for national estimates, as well as for estimates for the major subclasses: geographic regions, grades 10-12 enrollment, locale, and percentage of minority enrollment.

Response Rates

In October 1993, questionnaires (see appendix D) were mailed to 1,000 public secondary school principals. Principals were asked to have the questionnaire completed by the person in their school who was most knowledgeable about the school's academic curriculum and policies regarding the assignment of students to courses. Ten schools were found to be out of scope (no longer at the same location or serving the same population), leaving 990 eligible schools in the sample.

Telephone followup of nonrespondents was initiated in early December; data collection was completed by early March 1994 with 912 schools completing the survey. Of these, 703 schools (77 percent) completed the mailed questionnaire; telephone followup was conducted with the remaining 209 schools (23 percent).

The survey response rate was 92 percent (912 schools divided by the 990 eligible schools in the sample). Item nonresponse ranged from 0.0 to .9 percent.

Sampling and Nonsampling Errors

The response data were weighted to produce national estimates. The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability.

The survey estimates are also subject to nonsampling errors that can arise because of nonobservation (nonresponse or noncoverage) errors, errors of reporting, and errors made in collection of the data. These errors can sometimes bias the data. Nonsampling errors may include such problems as the differences in the respondents' interpretation of the meaning of the questions; memory effects; misrecording of responses; incorrect editing, coding, and data entry; differences related to the particular time the survey was conducted; or errors in data preparation. While general sampling theory can be used in part to determine how to estimate the sampling variability of a statistic, nonsampling errors are not easy to measure and, for measurement purposes, usually require that an experiment be conducted as part of the data collection procedures or that data external to the study be used.

To minimize the potential for nonsampling errors, the questionnaire was pretested w principals and assistant survey and the survey principals like those who completed the survey. During the design of the pretest, an effort was made to check for consistency of interpretation of questions and to eliminate ambiguous items. The questionnaire and instructions were extensively reviewed by the National Center for Education Statistics. Manual and machine editing of the questionnaire responses were conducted to check the data for accuracy and consistency.

Cases with missing or inconsistent items were recontacted by telephone. Imputations for item nonresponse were not implemented, as item nonresponse rates were less than 1 percent (for nearly all items, nonresponse rates were less than 0.5 percent). Data were keyed with 100 percent verification.

Variances

The standard error is a measure of the variability of estimates due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of schools reporting that they offer differentiated courses but allow students open access to any course provided they have taken the required prerequisite(s) is 71 percent, and the estimated standard error is 1.7. The 95 percent confidence interval for the statistic extends from [71 - (1.7 times 1.96)] to [71 + (1.7 times 1.96)], or from 67.4 to 73.9.

Estimates of standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate.

The mean square error of the replicate estimates around the full sample estimate provides an estimate of the variance of the statistic (see Welter 1985, Chapter 4). To construct the replications, 40 stratified subsamples of the fill sample were created and then dropped one at a time to define 40 jackknife replicates (see Welter 1985, page 183). A proprietary computer program (WESVAR), available at Westat, Inc., was used to calculate the estimates of standard errors. The software runs under IBM/OS and VAX/VMX systems.

Top