Quick Response Information System (QRIS)

4. SURVEY DESIGN

Target Population

Data collected through FRSS surveys are representative at the national level, drawing from a universe that is appropriate for each study. The FRSS collects data from state education agencies, local education agencies, public and private elementary and secondary schools (e.g., principals, teachers, guidance counselors, library media center specialists), and public libraries.

Sample Design

Sample sizes are relatively small (usually about 1,200 to 1,800 respondents per survey, but occasionally larger) so that data collection can be completed quickly.

Efficient probability sampling designs are an integral part of the FRSS. For sectors that are surveyed frequently in FRSS (e.g., school districts and public schools), a general approach to sampling is designed and modified as necessary to meet the specific goals of the study. For example, stratified probability-proportionate-to-size (PPS) sampling designs are generally used to ensure that (a) estimates with specified levels of sampling precision can be obtained for key subgroups of interest, and (b) both categorical and quantitative variables can be estimated reliably. The size measure is generally the aggregate square root of enrollment in the substratum. The use of the square root of enrollment to determine the sample allocation is considered reasonably efficient for estimating unit-level (e.g., district or school) characteristics and quantitative measures correlated with enrollment.

For some of the less frequently surveyed sectors, it is desirable to select a sample that is tailored to the specific needs of the individual survey. This specialization is most efficient when pertinent data are available for sample selection purposes. Examples of situations that necessitate designing and drawing special– purpose samples include surveys that are restricted to a particular subgroup (e.g., districts with summer migrant education programs or adult literacy programs), surveys that require concurrent fielding of different questionnaires in the same sector (e.g., library services for children and young adults), and related surveys involving different sets of respondents that must be linked through an overlapping sample design (e.g., the three surveys on educational technology conducted in 2008–09 that linked districts, schools, and teachers).

FRSS surveys of state education agencies do not involve sampling since all state education agencies are included. Sampling procedures for the other FRSS populations are discussed below.

Local education agencies (public school districts). The sampling frame is typically the NCES Common Core of Data (CCD) Public Elementary and Secondary Agency Universe File. (For information on CCD, see the CCD chapter.) The following variables are often used for stratification or sorting within primary strata: categories of enrollment size, geographic region, metropolitan status (community type), and poverty status. Other variables, such as charter school agency status, may be used to improve the precision of overall estimates, and to ensure minimum sample sizes for the analytic domains of interest.

As an example, the sample for the FRSS survey Career and Technical Education Programs in Public School Districts: 2016–17 consisted of approximately 1,800 eligible public school districts with high school grades in the 50 states and the District of Columbia. The nationally representative sample was selected from the 2013–14 NCES Common Core of Data (CCD) Local Education Agency (LEA) Universe file, which was the most current file available at the time of selection. The sampling frame for this survey included 11, 394 eligible public school districts that were coded with a highest grade of instruction of 11 or 12 in the CCD LEA Universe file. For purposes of this study, an eligible public district was either (1) a regular school district, or (2) a nonregular district that was not federally operated and had at least one operating vocational education school that did not have shared instruction. Of the 11,394 eligible districts in the sampling frame, 11,340 were regular districts.

The district sampling frame was stratified by district enrollment size (less than 1,000; 1,000 to 2,499; 2,500 to 9,999; 10,000 to 24,999; 25,000 to 99,999; and 100,000 or more) and community type (city, suburban, town, and rural) to create 21 primary strata. Within stratum, districts were sorted by region (Northeast, Southeast, Central, and West) and poverty status (poverty equal to less than 10 percent; 10 to 19.99 percent; 20 to 29.99 percent; and 30 percent or more) prior to selection to induce additional implicit stratification. Within each primary stratum, districts were selected systematically using sampling rates that depended on the size classification of the district.

Public elementary and secondary schools. The sampling frame is typically the NCES CCD Public School Universe file. The following variables are often used for stratification or sorting within primary strata: instructional level, categories of enrollment size, community type, geographic region, and either categories of poverty status (based on eligibility for free or reduced-price lunch) or categories of percent minority enrollment.

As an example, the sample of schools for the FRSS survey of School Safety and Discipline: 2013-14 consisted of approximately 1,600 regular public elementary, middle, and high school/combined schools in the 50 states and the District of Columbia. The nationally representative sample was selected from the 2011-12 NCES Common Core of Data (CCD) Public School Universe file, which was the most current file available at the time of selection. The sampling frame included 50,807 regular elementary schools, 16,536 regular middle schools, and 19,247 regular high school/combined schools. For purposes of this study, "regular" schools included charter schools. Excluded from the sampling frame were schools with a high grade of prekindergarten, kindergarten, or ungraded, schools with zero, missing, or "not applicable" enrollment, along with special education, vocational, and alternative/other schools, and schools outside the 50 states and the District of Columbia.

For this survey, the public school sampling frame was stratified by instructional level (elementary, middle, and high school/ combined), community type (city, suburban, town, and rural), and enrollment size (less than 300, 300 to 499, 500 to 999, and 1,000 or more) to create 45 primary strata. Within each stratum, schools were sorted by region (Northeast, Midwest, South, and West) and percent White, non-Hispanic enrollment in the school (missing, 96 percent or more, 81 to 95 percent, 51 to 80 percent, and 50 percent or less) prior to selection to induce additional implicit stratification. Within each primary stratum, schools were selected systematically using sampling rates that depended on the size classification of the school.

Private elementary and secondary schools. For this population, FRSS survey samples are constructed from the NCES Private School Universe Survey (PSS). (For information on PSS, see the PSS chapter.) The sample usually consists of regular private elementary, secondary, and combined schools, with a private school being defined as a school not in the public system that provides instruction for any of grades K–12 (or comparable ungraded levels) where the instruction is not provided in a private home. The following variables may be used for stratification or sorting within primary strata: instructional level (elementary, secondary, and combined), affiliation (Catholic, other religious, and nonsectarian), school size, geographic region, community type, and categories of percent minority enrollment. Schools are generally selected from each primary stratum with probabilities proportional to the weight reflecting the school’s probability of inclusion in the area sample.

Elementary and secondary school teachers. Teacher surveys generally use a two-stage sampling process. This involves selecting a sample of schools during the first stage (according to procedures described above) and obtaining lists of teachers from the selected schools. During the second stage of sampling, teachers are selected from the lists provided by the schools. The sampling criteria for teachers are dependent on the needs of the specific study.

As an example, for the sample of teachers for the upcoming FRSS survey of Teachers’ Use of Technology for School and Homework Assignments, a total of 2,000 schools will be selected for the study. Roughly equal sample sizes will be allocated to the major instructional levels (primary, middle, and high) and a minimum of 150 schools will be allocated to the remaining “other” category. Within each category of instructional level, the specified number of sample schools will be distributed to the five enrollment size classes in proportion to the number of full-time equivalent (FTE) teachers in the size class.

After sorting the schools in the sampling frame by type of locale and poverty status category within each primary sampling stratum, the sample of schools will be selected with probabilities proportionate to the number of FTE teachers in the school. Participating schools will be requested to provide lists of their classroom teachers for subsequent sampling purposes. Eligible teachers are those with self–contained classrooms in grades 3 or higher (generally teaching in primary schools) and those with departmentalized classes in one or more of the core academic subjects (generally teaching in middle and high schools). On average, 2.4 teachers will be randomly selected from each participating school for a total teacher sample size of 4,000.

Public libraries. Public libraries have been surveyed by the FRSS in the past (e.g., survey on programs for adults in public library outlets). For any future survey of public libraries, a sample will be drawn from the most recent Public Library Survey (PLS) universe file, currently conducted by the Institute of Museum and Library Services. The specific sampling procedures will depend on the needs of the survey.

Special populations. Other sources may serve as sampling frames, depending on the needs of the survey. For example, for Participation of Migrant Students in Title I Migrant Education Program (MEP) Summer-Term Projects, the districts and other entities serving migrant students were selected from the U.S. Department of Education’s 1995–96 Migrant Education Program Universe data file.

Top

Data Collection and Processing

Most FRSS surveys are self-administered questionnaires where respondents are offered the option of completing the survey on paper (submitted by mail, fax, or email) or via the Web, with telephone follow-up for survey nonresponse and data clarification. On rare occasions, a few have been telephone surveys, including one that used random digit dialing techniques. FRSS questionnaires are pretested, and efforts are made to check for consistency in the interpretation of questions and to eliminate ambiguous items before fielding the survey.

For example, for the Career and Technical Education Programs in Public School Districts: 2016–17 survey, questionnaires and cover letters were mailed to the superintendent of each sampled district in January 2017. The letter stated the purpose of the study and requested that the questionnaire be completed by the person in the district most knowledgeable about career and technical education (CTE) programs for high school students. Respondents were asked to respond for the current 2016–17 school year and the summer of 2016. Respondents were offered options of completing the survey on paper or online. Telephone follow-up for survey nonresponse and data clarification was initiated in February 2017 and completed in June 2017.

The unweighted survey response rate was 87 percent and the weighted response rate using the initial base weights was 86 percent. The survey weights were adjusted for questionnaire nonresponse and the data were then weighted to yield national estimates that represent all eligible public school districts in the United States.

Estimation Methods

Weighting. The response data are weighted to produce national estimates. The weights are designed to adjust for the variable probabilities of selection and differential nonresponse. Ineligible units are deleted from the initial sample before weighting and analysis. In the case of two-stage sampling—for example, for teacher-level surveys—the weights used to produce national estimates are designed to reflect the variable probabilities of selection of the sampled schools and teachers and are adjusted for differential unit (teacher sampling list and questionnaire) nonresponse.

Imputation. Because item nonresponse in FRSS surveys is typically very low, only limited use of imputation is required. Missing data are imputed for the items with a response rate of less than 100 percent using a “hot-deck” approach to obtain a “donor” from which the imputed values are derived. Donors are identified by matching selected characteristics to the case with missing data (the recipient). For categorical items, the imputed value is simply the corresponding value from the donor. For continuous numerical items (e.g., number of instructional rooms with wireless Internet connections), an appropriate ratio (e.g., the proportion of instructional rooms with wireless Internet connections) may be calculated for the donor, and this ratio applied to available data (e.g., reported number of instructional rooms) for the recipient to obtain the corresponding imputed value.

For example, in the Career and Technical Education Programs in Public School Districts: 2016–17 survey, missing data were imputed for the 73 items with a response rate of less than 100 percent. The missing items were all categorical data, such as whether the district offered CTE programs to high school students at various locations. The missing data were imputed using a “hot–deck” approach to obtain a “donor” district from which the imputed values were derived. The matching characteristics included district enrollment size, community type, region, and poverty status. In addition, relevant questionnaire items were used to form appropriate imputation groupings. Once a donor was found, the imputed value was simply the corresponding value from the donor district.

Future Plans

The next planned FRSS survey is Teachers’ Use of Technology for School and Homework Assignments for the 2018–19 school year.

Top