What ICILS Measures
The International Computer and Information Literacy Study (ICILS), a computer-based international assessment of 8th-grade students, was first conducted in 2013 by the International Association for the Evaluation of Educational Achievement (IEA) and was conducted again in 2018. The ICILS assessment framework, updated for each assessment cycle, provides a description of the field and the constructs to be measured. It outlines the design and content of the measurement instruments and describes how measures generated by those instruments relate to the constructs. The ICILS 2018 framework includes two dimensions of an individual’s skills with information communications technologies.
The first dimension, computer and information literacy (CIL), focuses on understanding computer use, gathering information, producing information, and communicating digitally.
The second dimension, computational thinking (CT), focuses on conceptualizing problems and operationalizing solutions.
The ICILS 2023 framework, which is based on the ICILS 2018 framework, is currently being updated to guide the development of ICILS 2023.
ICILS 2018 was conducted to help answer the following research questions, which apply to both the CIL and CT components:
- What variations in students’ CIL and/or CT skills exist within and across countries?
- What aspects of schools and countries are related to students’ CIL and/or CT skills?
- What are the relationships between students’ levels of access to, familiarity with, and self-reported proficiency in using computers and their CIL and/or CT skills?
- What aspects of students’ personal and social backgrounds (such as their gender and socioeconomic background) are related to their CIL and/or CT skills?
- What is the association between students’ CIL and CT skills?
Computer and Information Literacy (CIL)
is defined in the ICILS 2018 Assessment Framework
as “an individual’s ability to use computers to investigate, create, and communicate in order to participate effectively at home, at school, in the workplace, and in society” (Fraillon et al. 2018).
This definition relies on and brings together technical competence (computer literacy) and intellectual capacity (conventional literacies, including information literacy) to achieve a highly context-dependent communicative purpose that presupposes and transcends its constituent elements.
For more information on CIL, see the ICILS 2018 Assessment Framework
CLOSE
The CIL proficiency levels were established in 2013 after consideration of the content and difficulty of the test items. The item content and relative difficulty were analyzed to identify themes of content and process that could be used to characterize the different ranges, or levels, on the CIL achievement scale. This process was performed iteratively until each level showed distinctive characteristics, and the progression from low to high achievement across the levels was clear. The level boundaries—set at 407, 492, 576, and 661 scale points out of 700 total—form four proficiency levels. Student scores below 407 scale points indicate CIL proficiency below the lowest level targeted by the assessment instrument. The CIL proficiency levels did not change from 2013 to 2018.
CLOSE
CIL assessment modules include the following essential features:
- Students complete tasks solely on computer.
- The tasks have a real-world cross-curricular focus.
- The tasks combine technical, receptive, productive, and evaluative skills.
- The tasks reference safe and ethical uses of computer-based information.
Each CIL test module is comprised of a set of questions and tasks based on a real-world theme and following a linear narrative structure. Each module has a series of smaller discrete tasks,11 each of which typically takes less than a minute to complete. The narrative of each module positions the smaller discrete tasks as a mix of skill execution and information management tasks that students need to do in preparation for completion of a larger task. The larger task in each module typically takes 15-20 minutes to complete.
CLOSE
Computational Thinking (CT), new in ICILS 2018,
is defined as “an individual’s ability to recognize aspects of real-world problems that are appropriate for computational
formulation and to evaluate and develop algorithmic solutions to those problems so that the solutions could be operationalized
with a computer” (Fraillon et al. 2018). Computational thinking is the style of thinking used when programming a computer or
developing an application for another type of digital device. However, its usefulness extends beyond this. The reasoning strategies that underlie computational thinking can help make sense of complex ideas and solve problems. The assessment of computational thinking
through ICILS is an opportunity to gather international comparative data about how students are developing computational problem-solving skills in school.
For more information on the ICILS framework, see the ICILS 2018 Assessment Framework.
CLOSE
The CT performance regions were created using a different method. CT items were ordered by difficulty, then divided into three groups with equal numbers of items in each group. The descriptions of each region are syntheses of the common elements of students’ CT knowledge, skills, and understanding described by the items within each region. There are three regions in the CT scale: lower (below 459 scale points out of 700 total), middle (between 459 and 589 scale points inclusive), and upper (above 589 scale points). The regions of the CT scale should not be directly compared to the levels of the CIL scale.
CLOSE
CT assessment modules share the same essential features as CIL modules, but have an additional focus on systems thinking and development of algorithmic solutions. The 2018 CT assessment contained two 25-minute test modules. In countries participating in the CT option, students completed the two CT test modules in randomized order after they had completed the CIL test and the student questionnaire. The first CT test module (automated bus) focused on CT Strand 1: Conceptualizing problems; and the second (farm drone) focused on CT Strand 2: Operationalizing solutions. The automated bus module comprised a set of discrete, thematically related tasks relating to the configuration of the navigation and braking systems in a driverless bus. The farm drone module provided a visual coding interface that students were required to use to complete discrete coding tasks. The code in each task controlled the behavior of a programmable drone that could complete a set of actions on a farm. Students were presented with a workspace, draggable commands, and a visual output that showed the outcomes of the executed commands. The complexity of each task related to the number of targets and actions required to solve the problem instance.
CLOSE
Questionnaires
The international versions of the ICILS 2018 student, teacher, school and Information and Computer Technology (ICT)
Coordinator questionnaires
are included in the ICILS IDB User Guide, which is publicly available through the IEA data repository.
Several questions in the student questionnaire were adapted to be appropriate in the U.S. educational and cultural context,
and several U.S.-specific questions, such as race/ethnicity, were added to the international versions of the questionnaires.
The U.S. versions of the student, teacher, school, and ICT coordinator questionnaires are below.
In addition to the student assessment, students complete a questionnaire about their
- background;
- access to ICT;
- use of ICT; and
- familiarity with ICT at home and at school.
2018 Questionnaire
Information is also collected from 8th-grade students’ teachers. Teachers complete a questionnaire to collect information on their
- background;
- use of ICT;
- attitudes toward using ICT in teaching; and
- professional development related to ICT.
2018 Questionnaire
In addition, principals complete a questionnaire to collect information on school characteristics and broad policies, procedures, and priorities for ICT in the school.
2018 Questionnaire
In each school, the ICT coordinator completes a questionnaire about ICT resources in the school and the policies and practices that make use of those resources to support learning.
2018 Questionnaire
National Context
Lastly, a national context survey is completed by the national study centers of participating countries to provide information on policies, curricula, resources, and practices related to CIL and CT education from a national perspective.
For ICILS 2018, a summary of the national contexts for CIL and CT education is found in chapter 2 of Preparing for Life in a Digital World.