Data Limitations

As with any study, there are limitations to ICILS data that researchers should take into consideration. Estimates produced using data from ICLS are subject to two types of error—nonsampling and sampling errors. Nonsampling errors can be due to errors made in collecting and processing data. Sampling errors can occur because the data were collected from a sample rather than a complete census of the population.

Nonsampling errors
Nonsampling error is a term used to describe variations in the estimates that may be caused by population coverage limitations, nonresponse bias, and measurement error, as well as data collection, processing, and reporting procedures. The sources of nonsampling errors are typically problems like unit and item nonresponse, differences in respondents' interpretations of the meaning of the survey questions, response differences related to the particular time the survey was conducted, and mistakes in data preparation.

Missing data
Missing data for survey questionnaires, administrative data, and student assessment items were identified by missing data codes provided by international data processing center during the data cleaning process for all participating countries. The codes differentiate not administered/missing by design from presented but not answered/invalid. The assessment items also include an additional missing code for not reached. An item was considered as presented but not answered/invalid if the respondent was expected to answer the item based on answers provided for other questions in the sequence but no response was given (e.g., no box was checked in the item which asked, “Are you a girl or a boy?” or an uninterpretable responses (e.g., multiple responses to a question calling for a single response) was given. The not administered/missing by design code was used to identify missing due to items not administered to the student (e.g., those items excluded from the student's test booklet because of the booklet design which rotates assessment blocks across booklets), or skip patterns in the teacher, or principal, or when an item is not logical that the respondent answer the question (e.g., when the opportunity to make the response is dependent on a filter question). Finally, assessment items that are not reached were identified by a string of consecutive no responses continuing through to the end of the assessment.

The three key reporting variables identified in the ICILS data for the United States—student sex, student race/ethnicity, and the percentage of students in the school eligible for free or reduced-price lunch (FRPL)—all have low rates of missing responses. The response rates for these variables exceed the NCES standard of 85 percent and so can be reported without notation. Furthermore, the FRPL variable missing responses for public schools were imputed by substituting values taken from the Common Core of Data (CCD) for the schools in question. FRPL variable is only available for public schools.

Sampling errors
Sampling errors arise when a sample of the population, rather than the whole population, is used to estimate some statistic. Different samples from the same population would likely produce somewhat different estimates of the statistic in question. This fact means that there is a degree of uncertainty associated with statistics estimated from a sample. This uncertainty is referred to as sampling variance and is usually expressed as the standard error of a statistic estimated from sample data. The approach used for estimating standard errors in ICILS was jackknife repeated replication (JRR). Standard errors can be used as a measure for the precision expected from a particular sample. Standard errors for all of the reported estimates are included in each of the downloadable Excel tables that accompany each online figure and table at https://nces.ed.gov/surveys/icils/icils2018/theme1.asp. Scroll to the bottom of each web page for the link to the downloadable Excel table.

Although not presented in this report, confidence intervals provide another way to make inferences about population statistics in a manner that reflects the sampling error associated with the statistic. The intervals are calculated with a set confidence level, which defines the frequency that the population statistic will fall within the interval. All ICILS significance tests presented in this report use a p value of 0.05, which is equivalent to a 95 percent confidence level. Using that confidence level and assuming a normal distribution, the population value of this statistic can be inferred to lie within the confidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population. The endpoints of a 95 percent confidence interval can be calculated from the sampled mean and standard error. The lowest endpoint of the interval equals the mean minus the product of 1.96 times the standard error, while the highest endpoint of the interval equals the mean plus the product of 1.96 times the standard error. See Statistical Procedures section.