Appendix A. Methodology

Fast Response Survey System

The Fast Response Survey System (FRSS) was established in 1975 by the National Center for Education Statistics (NCES), U.S. Department of Education. FRSS is designed to collect issue-oriented data within a relatively short timeframe. FRSS collects data from state education agencies, local education agencies, public and private elementary and secondary schools, public school teachers, and public libraries. To ensure minimal burden on respondents, the surveys are generally limited to three pages of questions, with a response burden of about 30 minutes per respondent. Sample sizes are relatively small (usually about 1,000 to 1,500 respondents per survey) so that data collection can be completed quickly. Data are weighted to produce national estimates of the sampled education sector. The sample size permits limited breakouts by classification variables. However, as the number of categories within the classification variables increases, the sample size within categories decreases, which results in larger sampling errors for the breakouts by classification variables.

Sample Selection for Survey on High School Guidance Counseling

The sample for the FRSS survey on high school guidance counseling consisted of 1,001 secondary schools in the 50 states and the District of Columbia. It was selected from the 1999-2000 NCES Common Core of Data (CCD) Public School Universe file, which was the most current file available at the time of selection. The sampling frame included 16,897 regular secondary schools. For the purposes of the study, a secondary school was defined as a school with a highest grade of 11 or 12. Excluded from the sampling frame were schools with a highest grade lower than 11, along with special education, vocational, and alternative/other schools, and schools in the U.S. territories.

The public school sampling frame was stratified by enrollment size (less than 300, 300 to 499, 500 to 999, 1,000 to 1,499, and 1,500 or more), and poverty concentration as defined by the percentage of students eligible for free or reduced-price lunch (less than 35, 35 to 49, 50 to 74, and 75 to 100 percent).²⁷ Stratification by poverty concentration was designed to ensure a higher proportion of high poverty schools (i.e., 75 percent or more eligible for free or reduced-priced lunch) were selected, for analysis purposes.²⁸ These schools were therefore oversampled, resulting in low poverty schools (i.e., less than 35 percent eligible for free or reduced-priced lunch) being sampled at a reduced rate, in order to maintain the desired sample size. Finally, schools in the sampling frame were sorted by type of locale (city, urban fringe, town, rural) and region (Northeast, Southeast, Central, West) to induce additional implicit stratification.²⁹ These variables are defined in more detail below in the section Definitions of Analysis Variables.

Respondent and Response Rates

Questionnaires and cover letters for the study were mailed to the principal of each sampled school on January 29, 2002.³⁰ The letter introduced the study and requested that the questionnaire be completed by the school's lead guidance counselor or other staff member who is responsible for providing counseling services at the school. Telephone follow up was initiated February 19, 2002, and continued through May 2002 with schools that did not respond to the initial questionnaire mailing.

Of the 1,001 schools in the sample, 13 were found to be ineligible for the survey because they did not have an 11th or 12th grade. Another 31 were found to be ineligible because the school did not meet some other criteria for inclusion in the sample (e.g., it was an alternative education school). This left a total of 957 eligible schools in the sample. Completed questionnaires were received from 888 schools, or 93 percent of the sampled schools (Table A-1). The weighted response rate was 94 percent. The weighted item nonresponse rates ranged from 0 percent to 1.4 percent. Imputation for item nonresponse was not implemented. The weighted number of eligible institutions in the survey represent the estimated universe of regular secondary schools in the 50 states and the District of Columbia (Table A-1). The estimated number of schools in the survey universe decreased from the 16,944 schools in the CCD sampling frame to an estimated 15,789 because some of the schools were determined to be ineligible for the FRSS survey during data collection.

Sampling and Nonsampling Errors

The responses were weighted to produce national estimates (see Table A-1). The weights were designed to adjust for the variable probabilities of selection and differential nonresponse. The findings in this report are estimates based on the sample selected and, consequently, are subject to sampling variability. General sampling theory was used to estimate the sampling variability of the estimates and to test for statistically significant differences between estimates (see the Variances section below).

Variances

The standard error is a measure of the variability of an estimate due to sampling. It indicates the variability of a sample estimate that would be obtained from all possible samples of a given design and size. Standard errors are used as a measure of the precision expected from a particular sample. If all possible samples were surveyed under similar conditions, intervals of 1.96 standard errors below to 1.96 standard errors above a particular statistic would include the true population parameter being estimated in about 95 percent of the samples. This is a 95 percent confidence interval. For example, the estimated percentage of public high schools with written plans for their guidance programs is 61.2 percent and the standard error is 1.9 percent (see Tables 3 and 3a). The 95 percent confidence interval for the statistic extends from [61.2 + (1.9 x 1.96)] to [61.2 + (1.9 x 1.96)], or from 57.5 to 64.9 percent. The 1.96 is the critical value for a statistical test at the 0.05 significance level (where 0.05 indicates the 5 percent of all possible samples that would be outside the range of the confidence interval).

Because the data from the FRSS guidance counselor survey were collected using a complex sampling design, the variances of the estimates from this survey (e.g., estimates of proportions) are typically different from what would be expected from data collected with a simple random sample. Not taking the complex sample design into account can lead to an underestimate of the standard errors associated with such estimates. To generate accurate standard errors for the estimates in this report, standard errors were computed using a technique known as jackknife replication. As with any replication method, jackknife replication involves constructing a number of subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The mean square error of the replicate estimates around the full sample estimate provides an estimate of the variance of the statistic. To construct the replications, 50 stratified subsamples of the full sample were created and then dropped 1 at a time to define 50 jackknife replicates. A computer program (WesVar) was used to calculate the estimates of standard errors. WesVar is a stand-alone Windows application that computes sampling errors from complex samples for a wide variety of statistics (totals, percents, ratios, log-odds ratios, general functions of estimates in tables, linear regression parameters, and logistic regression parameters).

The test statistics used in the analysis were calculated using the jackknife variances and thus appropriately reflect the complex nature of the sample design. Bonferroni adjustments were also made to control for multiple comparisons where appropriate. For example, for a comparison involving g pairwise comparisons, each difference was tested at the 0.05/g significance level to control for the fact that g differences were simultaneously tested. The Bonferroni adjustment results in a more conservative critical value for judging statistical significance. This means that comparisons that would have been significant with a critical value of 1.96 may not be significant with the more conservative critical value.

When comparing percentage or ratio estimates across a family of three or more ordered categories (e.g., categories defined by percent minority enrollment), regression analyses were used to test for trends rather than a series of paired comparisons. For percentages, the analyses involved fitting models in WesVar with the ordered categories as the independent variable and the (dichotomous) outcome of interest (e.g., whether or not the school had written plans for guidance programs) as the dependent variable. For testing the overall significance, an analysis of variance (ANOVA) model was fitted by treating the categories of the independent variables as nominal categories. For the trend test, a simple linear regression model was used with the categories of the independent variable as an ordinal quantitative variable. In both cases, tests of significance were performed using an adjusted Wald F-test (WESVAR 4.0 User's Guide, page C-21). The test is applicable to data collected through complex sample surveys and is analogous to F tests in standard regression analysis. For estimated ratios, similar tests of overall significance and linear trends were performed using procedures analogous to those described in Chapter 12 of the book Analysis of Complex Surveys (Pfeffermann and La Vange 1989). A test was considered significant if the p-value associated with the statistic was less than 0.05.

Definitions of Analysis Variables

Enrollment Size - This variable indicates the total number of students enrolled in school based on data from the 1999-2000 CCD.³¹ The variable was collapsed into the following three categories:

Less than 500 students
500 to 1,199 students
1,200 or more students

School locale - This variable indicates the type of community in which the school is located, as defined in the 1999-2000 CCD (which uses definitions based on U.S. Census Bureau classifications). This variable was based on the eight-category locale variable from CCD and recoded into a four-category analysis variable for this report. Large and midsize cities were coded as city, the urban fringes of large and midsize cities were coded as urban fringe, large and small towns were coded as town, and rural areas outside and inside Metropolitan Statistical Areas (MSAs) were coded as rural. The categories are described in more detail below.

City - A large or midsize central city of a Consolidated Metropolitan Statistical Area (CMSA) or Metropolitan Statistical Area (MSA).

Urban fringe - Any incorporated place, Census-designated place, or non-place territory within a CSMA or MSA of a large or midsize city, and defined as urban by the Census Bureau.

Town - Any incorporated place or Census-designated place with a population greater than or equal to 2,500 and located outside a CMSA or MSA.

Rural - Any incorporated place, Census-designated place, or non-place territory defined as rural by the Census Bureau.

Percent College Bound - This variable represents the percentage of public high school 2000-2001 graduates who were reported by schools as enrolling in a 2-year or 4-year college directly after high school, as reported by the school. It was derived for each school by summing the percentages from column A of questions 12a and 12b (i.e., the percentage of students who enrolled in a 4-year college and the percentage that enrolled in a community college or other less-than-4-year postsecondary education institution). Data on this variable were missing for six schools; missing data were excluded from all analyses by questionnaire variables. The variable was collapsed into the following three categories:

Less than 50 percent of students
50 to 74 percent of students
75 percent or more of students

Vocational Courses per 100 Students - This variable indicates the ratio of vocational courses offered per 100 grade 9-12 students. It was derived from question 11 of the 2001 FRSS survey on high school guidance counseling, and the total number of high school students (grades 9 through 12) reported in the 1999-2000 CCD School Universe Survey. Vocational courses were defined on the cover of the questionnaire, and include courses available to students at the responding schools or at area or regional vocational schools. Data on this variable were missing for 14 schools; missing data were excluded from all analyses by questionnaire variables. The variable was collapsed into the following three categories:

Fewer than 3 courses
3 to 6 courses
More than 6 courses

Region - This variable classifies schools into one of the four geographic regions used by the Bureau of Economic Analysis of the U.S. Department of Commerce, the National Assessment of Educational Progress, and the National Education Association. Data were obtained from the 1999-2000 CCD School Universe file. The geographic regions are:

Northeast - Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, and Vermont

Southeast - Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia

Central - Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin

West - Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming

Percent Minority Enrollment - This variable indicates the percentage of students enrolled in the school whose race or ethnicity is classified as one of the following: American Indian or Alaska Native, Asian or Pacific Islander, non-Hispanic Black, or Hispanic, based on data in the 1999-2000 CCD School Universe file. Data on this variable were missing for eight schools; missing data were excluded from all analyses by questionnaire variables. The variable was collapsed into the following four categories:

Less than 6 percent minority
6 to 20 percent minority
21 to 49 percent minority
50 percent or more minority

Access to Area/Regional Vocational School - This variable indicates whether any vocational education courses (as defined on the cover of the questionnaire) were offered to students at the sampled high school through an area or regional vocational school. The variable was derived from part (b) of question 11 of the 2001 FRSS survey on high school guidance counseling. Thus, if the response to question 11 (b) was greater than zero courses, the school was coded as having access to an area/regional vocational school. Data on this variable were missing for 13 schools; missing data were excluded from all analyses by questionnaire variables. This is a dichotomous variable:

Has access to area/regional vocational school
Does not have access to area/regional vocational school

It is important to note that many of these school characteristics may be related to each other. For example, school size and locale are related, with city schools typically being larger than rural schools. Other relationships between these analysis variables may exist. However, this E.D. Tab report focuses on bivariate relationships between the analysis variables and questionnaire variables rather than more complex analyses.

The 1984 Supplement to High School and Beyond Data

Data from the 1984 supplement to HS&B on guidance counseling were collected through the guidance questionnaire of the Administrator and Teacher Surveys (ATS) conducted as part of the second followup of the High School and Beyond (HS&B) surveys.³² The questionnaire was designed to be completed by the schools? heads of guidance counselors. Data from the 1984 supplement to the HS&B guidance questionnaire were re-analyzed for three questionnaire items that are the same or similar to items from the 2002 FRSS survey on high school guidance counseling. The three questionnaire items from the 1984 supplement to the HS&B guidance questionnaire are shown in appendix C.

To create a comparable sample of 1984 high schools, a subset of schools was selected from the 1984 supplement to the HS&B guidance survey that was similar to the schools sampled for FRSS; that is, regular public high schools in the 50 states and the District of Columbia. As with the FRSS study, a high school was defined as those with a highest grade of 11 or 12. Excluded from the sample were schools with a highest grade lower than 11, private schools, special education schools, vocational and alternative/other schools, and schools in the U.S. territories. Of the 537 public and private schools that responded to the guidance questionnaire of the 1984 supplement to HS&B, 319 regular public high schools with a high grade of 11 or 12 were selected for comparisons with the FRSS survey.

Variance Estimation for HS&B Data

The original variance estimation for the HS&B used the Taylor series approximation method. However, the stratum codes that are needed to implement this method were not available in the public use files. Therefore, to obtain approximate variances for the HS&B sample, jackknife replicates were used to calculate the required standard errors. As with the FRSS analyses, jackknife replicates were derived by sorting the schools in the HS&B sample by geographic region (nine Census divisions), degree of urbanization (urban central city, suburban, and rural), income level of the community (measured by percentage of disadvantaged students), percentage of non-English speaking students, proximity to a college, and 12th-grade enrollment size. Jackknife replicates were then systematically assigned to the schools in the sorted list to 50 subsamples.

The sorting specified above to create the jackknife replicates for the HS&B sample is rough because not all of the original stratifiers were available in the public use files. The original sample of schools was drawn based on the following stratification: (1) type of control (public, Catholic, non- Catholic private), (2) geographic region (nine Census divisions), (3) racial and ethnic composition (various combinations of White, non-Hispanic, Black, and Hispanic enrollment ratios), (4) degree of urbanization (urban central city, suburban, and rural), (5) income level of the community, (6) proximity to a college, and (7) enrollment size. However, income level, enrollment size, and the racial and ethnic composition variables were missing in the public use file. In place of these, percentage of disadvantaged students, 12th-grade enrollment size, and percentage of non-English-speaking students (or students not speaking English at home) were used as described above.

Comparisons Between FRSS and HS&B Data

Data from the 1984 supplement to HS&B were compared with the 2002 FRSS data for three comparable items (see appendix C). There are a number of possible reasons why these data sets might yield different estimates on these items. One obvious reason is that the differences show actual change between 1984 and 2002. However, it is important to consider other possibilities. While the subset of schools from HS&B was very similar to the FRSS sample of schools, there may still be some differences in the samples for the two surveys that result in differences in estimates. In addition, the FRSS questionnaire and the HS&B questionnaire provided different response contexts for guidance counselors. Whereas the FRSS questionnaire contained 3 pages of questions and collected information in a very compact format, the HS&B questionnaire had more than 62 questions, of which only 3 were included for comparisons with FRSS data.

FRSS Survey Sponsorship and Acknowledgments

The survey was performed under contract with Westat. Bernie Greene was the NCES Project Officer. The data were requested by the Office of Vocational and Adult Education and the National Center for Education Statistics of the U.S. Department of Education.

This report was reviewed by the following individuals:

Outside NCES

Steve Equall, Planning and Fiscal Management for Career and Technical Education, Nebraska
• Gisela Harkin, Office of Vocational and Adult Education, U.S. Department of Education
   
• Carolyn Lee, Office of Vocational and Adult Education, U.S. Department of Education
   
• Stephanie Cronen, American Institutes for Research, Education Statistics Services Institute
   
• Lawrence Lanahan, American Institutes for Research, Education Statistics Services Institute
   
• Sally Dillow, American Institutes for Research, Education Statistics Services Institute
   
• Kimberley Green, National Association of State Directors of Vocational-Technical Education Consortium
   
• Daniel Goldenberg, Policy and Program Studies Service

Inside NCES

• William Hussar, Early Childhood, International, and Crosscutting Studies Division
   
• Val Plisko, Early Childhood, International, and Crosscutting Studies Division
   
• John Ralph, Early Childhood, International, and Crosscutting Studies Division
   
• Kathryn Chandler, Elementary/Secondary and Libraries Studies Division
   
• Patrick Rooney, Early Childhood, International, and Crosscutting Studies Division
   
• Lance Ferderer, Assessment Division
   
• Marilyn Seastrom, Statistical Standards Program, Office of the Deputy Commissioner
   
• Bruce Taylor, Statistical Standards Program, Office of the Deputy Commissioner

---

²⁷The CCD data for enrollment size were missing for 47 schools; these schools were excluded from the sampling frame. In addition, data for the number of students eligible for free or reduced-price lunch were missing for 504 schools; these schools were assigned to a separate stratum for sampling purposes.

²⁸Schools were not analyzed by poverty concentration, however, due to a relatively high proportion of missing data for responding schools.

²⁹There were no missing data for type of locale and region in the sampling frame.

³⁰The survey was developed and approved by the Office of Management and Budget (OMB) in fall 2001.

³¹Schools with missing enrollment (47 cases) were excluded from the sampling frame because they were primarily out of scope for the survey (e.g., alternative education or special education schools).

³²Base year data for HS&B were collected in 1980, and the first follow up was conducted in 1982. In 1984, the ATS was conducted as a supplemental survey in approximately half of the schools sampled in the original HS&B study. Within the ATS, separate questionnaires were administered to high school teachers, administrators, vocational education coordinators, and heads of guidance programs. For a description of the HS&B methodology see U.S. Department of Education, National Center for Education Statistics. (1981). High School and Beyond: A National Longitudinal Study for the 1980's. Sample Design Report, by Martin Frankel, Luane Kohnke, David Buonanno, and Roger Tourangeau. For a description of ATS, see National Center for Education Statistics, High School and Beyond, 1980: Sophomore and Senior Cohort Second Follow-Up (1984), Vol. IV (ICPSR 8443) (Washington, DC: U.S. Department of Education, 1989).

Top