Teaching and Learning International Survey (TALIS)

4. SURVEY DESIGN

The survey design for TALIS data collections is discussed in this section.

Target Population

The desired TALIS target population consisted of schools where ISCED Level 2 education is provided along with the affiliated principals and teachers. No subject matter was excluded from the scope of TALIS teachers. Thus, coverage of TALIS extended to all teachers of ISCED Level 2 and to the principals of the schools where they teach.

The formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching. An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the United States, ISCED Level 2 teachers are those who provide any instruction for grades 7, 8, and/or 9. Teachers who taught a mixture of programs at different levels including ISCED Level 2 programs in the target school were included in the TALIS universe, as well as teachers who engaged with individual or small groups of students in “pull in” or “push out” programs. There was no minimum cut-off for how much ISCED Level 2 teaching- that is, either the number of classes or students- these teachers need to be engaged in to be included.

Some 140 U.S. schools participated in the U.S. TALIS 2013 survey; 111 principals and 2,034 teachers completed questionnaires. Data collection occurred from March through May 2013.

International Sample Design

The sample design for TALIS 2013 was a stratified two-stage probability sample. This meant that teachers (second stage units or secondary sampling units, abbreviated as SSUs) were randomly selected from the list of in-scope teachers for each of the randomly selected schools (first-stage or primary sampling units, abbreviated as PSUs).

Although TALIS’s intent was to be as inclusive as possible, guidelines allowed for schools to be excluded for approved reasons (e.g. remote regions, very small schools, special needs only schools). TALIS also had set guidelines for teacher exclusions including: 1) substitute, emergency, or occasional teachers; 2) teachers of adults; 3) teachers on long-term leave; 4) teacher aides; 5) pedagogical support staff; and, 6) health and support staff. For more information, please refer to TALIS 2013 U.S. Technical Report (NCES 2015-010).

International technical standards for the TALIS 2013 main study required that the teacher sample size be a minimum of 3,400 surveyed ISCED Level 2 teachers for the main study, or the National Defined Target Population. The school sample size must be a minimum of 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.

The minimum number of teachers required within each sampled school is suggested to be 20 to allow for reliable estimation and modeling, while allowing for some amount of nonresponse. In schools where fewer than 20 teachers of ISCED Level 2 are found, all will be in the sample. In schools where the number of teachers of ISCED Level 2 is between 21 and 30, it is suggested that all the available teachers be sampled. However, each education system will have the choice to determine the sample size cutoff. The United States decided to select 22 teachers from any schools with 22 or more eligible teachers. This number was based on calculations which estimated the total number of TALIS eligible teachers at 201 sample schools, and anticipated a yield of at least 3,500 teachers (before refusals). Based on the experience from the previous TALIS, this would provide a sufficient level of precision for the analysis (after refusals).

Response Rate Targets

School response rates. The technical standards for the TALIS 2013 main study required that school response rates must be at least 75 percent of sampled schools. If a response rate was below 75 percent then an acceptable response rate can still be achieved through agreed upon use of substitute schools. Two substitute schools were preselected to replace each school in the sample. Although substitute schools could be called upon to replace nonresponding schools, education systems were encouraged to do all they can to obtain the participation of the schools in the original sample.

Education systems must obtain participation by 50 percent or more of the original sampled schools. Education systems that experienced less than 75 percent sample school participation after substitution have to demonstrate convincingly that their sample was not significantly biased. TALIS established three response rate zones-good, fair, or poor. “Good” meant the education system’s data were included in the international database. “Fair” meant that the education system’s data may not be recommended for full inclusion in international comparisons. “Poor” meant that the education system’s data were not included in the international comparisons. The TALIS Board of Participating Countries made the final decision on whether to include the education system’s data in international comparisons while taking into account various other factors.

Teacher response rates. A minimum response rate of 75 percent of sampled teachers in participating schools (original sample or substitute school) was required. Responding schools that yielded at least 50 percent of sampled teachers were considered as participating schools; schools that failed to meet that threshold were considered as “nonparticipating” even though the number of responding teachers may be enough to contribute to some of the analyses.

SAMPLE DESIGN IN THE UNITED STATES

The design of the U.S. school sample for TALIS 2013 was developed to follow international requirements as given in the TALIS 2013 Sampling Manual-Main Survey Version (OECD 2012). However, it is also worth noting that United States did not meet the international participation rate standards ². The TALIS 2013 U.S. sample was based on a stratified two-stage probability sample design. At the first stage the primary sampling units were individual ISCED Level 2 schools, selected systematically with probability proportional to size from the stratified sampling frame. At the second stage, the secondary sampling units were the in-scope teachers, selected randomly within the sample schools.

The U.S. school sampling frame was developed from two national databases in the National Center for Education Statistics– public schools in the Common Core of Data (CCD) and private schools in the Private School Universe Survey (PSS). These sources provide full coverage of all TALIS-eligible teachers in the education system in the United States. The TALIS school frame was constructed using the 2010-11 CCD and the 2009-10 PSS, the most current data at the time of the TALIS frame construction.

The sampling frame for the main study used two explicit strata: school control (i.e., public/private) and grade structure. The grade structure is defined with the following categories:

• Middle-Junior, which includes middle school (grades 6 to 8) or junior high (grades 7 to 9, or grades 7 and 8);
• High school (grades 9 to 12); and
• Other (any other grade structure that includes at least one ISCED Level 2 grade).

The sampling specifications for selecting the schools for the main study specified the following three implicit stratification variables: (1) region (Northeast, Midwest, South, and West), (2) percent minority students, and (3) number of ISCED Level 2 teachers (measure of size). Within each explicit stratum the schools were sorted by a hierarchical combination of the implicit stratum variables in order to improve the representativeness of the sample across these variables.

The sample schools were allocated to the different explicit strata proportionally to the total number of ISCED Level 2 teachers. Given the small proportion of the schools in the combined private middle-junior and high school stratum, the proportional allocation for this combined stratum was increased from 3 to 4 schools, resulting in a final sample of 201 schools. During the data collection, three schools were found to be out-of-scope, reducing the sample to 198 schools. Per international guidelines, any school declining to participate is replaced by a pre-selected similar school.

To allow for reliable estimation and modeling, while taking into account the expected levels of nonresponse, the sample size for the U.S. TALIS main study was set at 22 ISCED Level 2 teachers within each participating school, or all of the eligible teachers when the school had 22 or fewer. In schools with more than 22 eligible teachers, a random sample of 22 eligible teachers was drawn. The distribution of eligible teachers at eligible schools is an estimate since teacher lists were not available.

School coordinators were asked to provide lists of all eligible teachers in the school (using a standardized Teacher Listing Form). To reduce burden, a Teacher Listing Form was provided to the school coordinators both in hard copy and in electronic form.

Once the Teacher Listing Form was received from a school, it was formatted for importing into WinW3S, the sampling software developed by the International Association for the Evaluation of Educational Achievement (IEA) and provided by OECD for use on this project. After importing the list from a school, the appropriate validation checks were run, the teachers were sampled, and the Teacher Tracking Forms were output from WinW3S.

Data Collection and Processing

The TALIS Board of Participating Countries developed technical standards that provided standardized procedures for all education systems to follow. NCES was responsible for the implementation of TALIS in the United States in accordance with the international standards and procedures. TALIS 2013 data collection and associated tasks were carried out through a contract with Strategic Analytics, Inc. and its two subcontractors, Strategic Research Group, Inc. (SRG), and Sabre Systems, Inc. Strategic Analytics was responsible for project coordination, preparation of recruitment materials, preparation of the U.S. data files, and reporting. Sabre Systems was responsible for school and teacher sampling, data processing, and bias analyses. SRG was responsible for recruitment of schools and teachers, adaptation of the international instruments, and data collection. SRG worked closely with the school principal and a school coordinator (a school staff member designated by the principal) in conducting the data collection.

Reference dates. Each education system selected its own timeframe for survey administration, ranging from 12 days to 4 months, within the internationally prescribed time period of the end of the school year. The end of the school year was purposefully selected to guarantee comparability of collected data. During this period, principals and teachers were free to fill in the questionnaires whenever they chose. The overall target was 100% within-school participation.

Data collection. All data collection activities were conducted by mail, e-mail, and telephone. Quality control activities were performed by SRG and Strategic Analytics staff, as well as an international quality control monitor appointed by OECD.

Each participating school was required to designate a staff member to serve as school coordinator. School coordinators received a School Coordinator Manual to use in performing their activities. A significant portion of this document provided instruction on assembling a list of eligible teachers. The manual also covered distribution of the questionnaires, completing the Teacher Tracking Form, quality control that would be conducted during TALIS, and returning materials to SRG.

Beginning in February 2013, and continuing as schools agreed to participate, the school coordinators were contacted, and mailed and/or e-mailed an introductory letter along with the School Coordinator Manual and Teacher Listing Form. The Teacher Listing Form was offered as an Excel file delivered by e-mail, but was available on paper as well. SRG staff contacted school coordinators by telephone and e-mail to obtain the completed Teacher Listing Forms.

Following teacher sampling, SRG mailed the principal and teacher packets to the school coordinator, who was responsible for distributing them. SRG staff remained in contact with school coordinators by telephone and e-mail to encourage the completion of the questionnaires.

SRG received completed Teacher Listing Forms by mail or e-mail. Once received, they were reviewed for completeness and accuracy. One key check involved the number of teachers listed on the form. This was compared to an estimate of teachers from the sampling frame, and if the number differed by more than 25 percent, the school coordinator was contacted to resolve the discrepancy. As problems were discovered, school coordinators were asked to resubmit a corrected Teacher Listing Form.

Once the Teacher Listing Form was deemed to be complete and accurate, the data were entered into WinW3S, the sampling software provided by OECD. After importing the list from a school, the appropriate validation checks were run, the teachers were sampled, and the Teacher Tracking Forms were output from WinW3S. A total of approximately 2,630 teachers (an average of 19 per school) were sampled. In schools with 22 or fewer eligible teachers, all were selected; in schools with 23 or more eligible teachers, 22 were randomly selected.

Following sampling, SRG staff mailed the school coordinator materials needed for the data collection. As schools received these packages, data collection began. The first packages were sent at the beginning of March 2013. Because of the length of time it took to recruit many of the schools, and in some cases, receive completed Teacher Listing Forms, data collection could not be started until much later. Data collection did not begin in many schools until mid-way or very late into the data collection phase. For this reason as well as the continued push to recruit additional schools, the deadline for data collection was extended from April 2013, to May 2013, with the approval of OECD.

SRG staff continued to contact schools on a regular basis throughout the data collection period. The first follow-up calls began in March 2013. Subsequently, the school coordinator was called and/or e-mailed at least once a week. These contacts continued until all sampled teachers had responded or data collection ended. From mid-April through May, NCES staff also contacted schools to encourage participation.

Data processing. The data collection in the United States was led by the staff at SRG. The SRG staff were responsible for processing the Teacher Tracking Forms and entering them into the WinW3S software for teacher sampling. The primary data collection mode in the United States was through online instruments. The online instruments were administered using the Online Data Collection (ODC) software provided by the IEA-DPC, but that resided on an NCES server for the U.S. collection. Paper responses were entered and verified using the Data Management Expert (DME) software, also provided by the IEA-DPC. The data entry and verification steps consisted of SRG staff entering the paper responses, as well as managing the collection of the online and paper responses. In the case of paper responses, SRG staff entered and verified the data and, at the end of collection produced a DME file for both the teacher and school file.

The verification steps handled by SRG staff included an automatic validation of the paper surveys entered into the DME, as well as data checks that checked for duplicate codes and data output outside the expected valid range or values defined as valid. SRG staff reviewed the reports and verified that invalid entries had been correctly entered and that the available data corresponded to the expected based upon the participation indicators and entries on the tracking forms. The SRG staff provided the IEA-DPC staff with detailed documentation but did not make any changes to the data other than correcting data entry errors.

Estimation Methods

Weighting. The use of sampling weights is necessary for the computation of statistically sound, nationally representative estimates when using a complex survey sampling procedure. Survey weights adjust for the probabilities of selection for individual schools and teachers. TALIS used a stratified multi-stage probability sampling plan with unequal probabilities of selection. The school sampling included a probability proportional to size systematic sample, while the teacher sample was a simple random sample within selected schools. Survey weighting for all participating education systems was carried out by Statistics Canada, as part of the TALIS consortium.

School weights. The schools weights were a function of the school base weight, or design weight, and a nonresponse adjustment factor. School Base Weight is the probability of selection using the systematic random sampling scheme with probability proportional to size. Nonresponse Adjustment Factor is an adjustment that allocates the weight of the nonresponding schools to responding schools so that estimates reflected the population the sample was intended to represent.

Teacher weights. The teacher weighting was more complicated than the school weighting because, while it was a simple random sample at the school level, it included the school base weight as well as four additional adjustment factors. The final teacher weight adjusted for school nonresponse, teacher nonresponse, and incidental inclusions, and included a multiplicity adjustment. The school base weight incorporates the probability of selection of the school into the teacher weight and the nonresponse adjustments account for participation, or lack of participation, at each level. The incidental inclusion adjustment accounts for teachers who are also principals in the U.S. case. The multiplicity adjustment factor adjusts for the fact that teachers working in more than one ISCED Level 2 school had more chance of being selected in the sample.

School Base Weight is the probability of selection using the systematic random sampling scheme with probability proportional to size. School Nonresponse Adjustment is an adjustment that accounts for nonresponse at the school level. School nonresponse adjustments were applied within the explicit strata, reallocating the weight of nonresponding schools within each stratum to the responding schools.

Teacher Base Weight is the inverse of the probability of selection of the teacher at the time of selection. Teacher Nonresponse Adjustment is an adjustment that allocates the weight of the nonresponding teachers to responding teachers so that estimates reflected the population the sample was intended to represent. The teacher nonresponse adjustment included adjustments within each explicit strata that accounted for nonresponding teachers as well as teachers that left the school after having been selected for the sample.

Adjustment for Incidental Exclusions is an adjustment to account for teachers who are also principals in the U.S. case. Multiplicity Adjustment is an adjustment that accounts for the fact that teachers working in more than one ISCED Level 2 school had more chance of being selected in the sample.

Imputation. No imputation was conducted for TALIS 2013.

Measuring Trends. In an effort to continue to improve TALIS while simultaneously maintaining consistency, new directions for the study have been adopted for the TALIS 2013. At the same time, many of the TALIS 2008 themes, scales, and indicators have been preserved. This approach serves the dual purpose of allowing for the analysis of trends and permitting the investigation of contemporary issues in teaching and learning. More specifically, the TALIS questionnaires are designed to allow for some trend analysis between TALIS 2008, TALIS 2013 and further cycles of TALIS, while permitting for additional inquiry into areas identified as high priority by participating OECD countries, partner economies and sub-national entities. TALIS 2013 was the first time that TALIS was administered in the United States. Thus, there is not sufficient information available to measure trends in the U.S. until data from new cycles are available.

Future Plans

The next cycle of TALIS data collection will take place in 2018.

Top