Teaching and Learning International Survey (TALIS)

5. Data Quality and Comparability

A comprehensive program of continuous quality monitoring was central to ensuring full, valid implementation of the TALIS procedures and the recording of deviations from these procedures. National Project Managers (NPMs) and members of the TALIS 2013 Board of Participating Countries nominated suitably qualified individuals to carry out the international quality control monitoring program (IQCM), which was conducted externally to the national center. The IQCMs reviewed national TALIS materials to determine if the guidelines of the TALIS 2013 International Study Center (ISC) had been followed and visited schools to conduct standardized interviews on administering the survey with the School Coordinator (SC).

Despite the efforts taken to minimize error, as with any study, TALIS has limitations that researchers should take into consideration. This section contains a discussion of two possible sources of error in TALIS: sampling and nonsampling errors.

Sampling Errors

Estimating sampling errors when dealing with a complex design like TALIS must incorporate the survey design and unequal weights to obtain unbiased estimates. Not accounting for either may lead to significant underestimation of the sampling error. There are a number of methods that take into account the complex sample design and provide appropriate estimates of sampling errors. The Balanced Repeated Replication (BRR) method is used for TALIS and 100 replicate weights are provided for the implementation of this method in the estimation of standard errors for all analysis when using the appropriate software and commands. The TALIS 2013 Technical Manual (OECD 2014b) covers this in greater detail and the IEA International Database (IDB) Analyzer software, available on the Internet (http://www.iea.nl/data.html), uses the replicate weights to produce the appropriate standard errors when used in conjunction with SPSS.

Nonsampling Error

Nonsampling error is a term used to describe variations in the estimates that may be caused by population coverage limitations, nonresponse bias, and measurement error, as well as data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems such as unit and item nonresponse, the differences in respondents’ interpretations of the meaning of survey questions, response differences related to the particular time the survey was conducted, and mistakes in data preparation.

In general, it is difficult to identify and estimate either the amount of nonsampling error or how much bias it causes. In TALIS 2013, efforts were made to prevent such errors from occurring and to compensate for them when possible.

If there were no systematic differences among specific groups under study in their tendency to give socially desirable responses, then comparisons of the different groups would accurately reflect differences among groups. Readers should be aware that respondent bias may be present in this survey as in any survey; however, it is not possible to state precisely how such bias may affect the results.

Coverage error. The TALIS program of surveys aims to cover all teachers of a given ISCED level in a participating education system. For national reasons, participating education systems can choose to restrict the coverage of their national implementation of TALIS to parts of the education system. Ideally, all the members of a target population should be admissible to sampling and data collection. This is the option that TALIS chose. As a consequence, the international survey population (those who can be surveyed) is identical to the international target population (those who should be surveyed).

TALIS recognizes that attempting to survey teachers in very small schools, that is, schools with no more than three teachers at the ISCED level of interest, and those teaching in schools located in geographically remote areas tends to be costly, time consuming and statistically inefficient. Therefore, participating education systems can excuse those teachers from the TALIS data collection, thus creating a national survey population different from the national target population. TALIS 2013 therefore required the National Project Manager (NPM) for each education system to document the reasons for exclusion as well as the size, location, and clientele of each excluded school.

In TALIS 2013, the United States developed its school sampling frame from CCD and PSS. These sources provide full coverage of all TALIS–eligible teachers in the education system in the United States.

Top

Nonresponse error. Nonresponse error results from nonparticipation of schools, teachers, and principals. TALIS international requirements stipulate that the weighted school response rate target is a minimum of 75 percent (after substitution). A minimum of 50 percent of schools from the original sample of schools are required to participate for data to be included in the international database. Substitute schools are allowed to be used (selected during the sampling process) to increase the response rate. TALIS 2013 also requires a minimum participation rate of 50 percent of sampled teachers from each school in order for that school and its respondents to be included.

Unit nonresponse. One‑hundred fifty‑two schools were recruited to participate in TALIS 2013. One of these schools never identified a school coordinator, leaving 151 schools. A further 11 of these schools did not return their Teacher Listing Form, resulting in a final total of 140 participating schools. Of these, 122 schools had 50 percent or more response among teachers (78 original schools and 44 substitute schools). By the close of data collection close to 80 percent of principals and teachers responded.

Top

Unit nonresponse bias analysis. The TALIS 2013 standards also require that nonresponse bias analyses need to be conducted if weighted school response rates are less than 75 percent (after substitution). NCES statistical standards for surveys stipulate that a nonresponse bias analysis is required at any stage of data collection with a weighted unit response rate less than 85 percent (without substitution).

The investigation into nonresponse bias at the school level for the U.S. TALIS 2013 school sample showed that there was no statistically significant relationship detected between participation status and the school characteristics that were available for analysis. It also suggested that there was evidence that the use of substitute schools reduced the potential for bias, based on an examination of the relative bias between estimates across the variables examined here. The application of nonresponse adjusted weights appears to have reduced, but certainly not eliminated, the potential for bias as evidenced by the smaller measures of bias in most categories.

The investigation into nonresponse bias at the teacher level, which is the unit level of analytic interest in TALIS, revealed that two of the variables examined (school control and grade structure) showed statistically significant relationships with teacher participation when examining base‑weighted distributions. Based on the results of row‑level t‑tests, public school teachers were overrepresented among participating teachers in original schools while private school teachers were underrepresented among participating teachers. When taking into consideration all participating teachers at both original and substitute schools, and accounting for the nonresponse adjustments, these results did not hold. The multivariate results were consistent with the bivariate findings in most respects. Neither school control nor grade structure were significant in the multivariate setting, but the percent of minority students was significantly related to nonresponse in the regression model in spite of the nonsignificant results for the model.

Taken all together, the investigation of unit-level nonresponse in the U.S. TALIS sample revealed there is potential for nonresponse bias in some estimates at the school and teacher level, although the amount of bias varied greatly depending on the unit level (school or teacher) and the variable being examined.

Item nonresponse bias analysis. NCES standards require nonresponse bias analysis when item-level analysis for all items with an item-level response rate below 85 percent. The item-level nonresponse bias analysis was limited to the single item with less than an 85 percent response rate that required analysis, item 24O2 in the teacher questionnaire. The analysis of the item on professional development in the area of implementation of national/state curriculum standards showed evidence of potential bias, particularly with respect to several categories of age and experience. There was little evidence of bias with respect to gender and full-time teaching status, but part-time teachers were less likely to respond to this item. Care should be taken when analyzing this item, particularly with respect to the variables that showed evidence of potential bias.

Measurement error. Measurement error is introduced into a survey when its instruments do not accurately measure the knowledge or aptitude they are intended to assess.

Data Comparability

The U.S. TALIS 2013 data may require confirmation of the estimates using other data sources, such as the Schools and Staffing Survey (SASS), when possible.

For example, when conducting unit nonresponse bias analysis, further evidence of potential bias in the U.S. TALIS teacher sample came from a comparison to a similar sample of teachers in the Schools and Staffing Survey (SASS). Based on comparisons of a limited number of key demographic characteristics shared between the two studies, the U.S. TALIS teacher sample appears to overrepresent teachers who report a full-time contract status and those that have the most number of years of teaching experience (i.e., 10+ years) while it underrepresents teachers who report a part-time contract status and those with the fewest years of teaching experience (i.e., less than 4 years).

 

Top