What ICILS Measures
The International Computer and Information Literacy Study (ICILS), a computer-based international assessment of 8th-grade students, was first conducted in 2013 by the International Association for the Evaluation of Educational Achievement (IEA) and was conducted again in 2018 and 2023. The ICILS assessment framework, updated for each assessment cycle, provides a description of the field and the constructs measured. It outlines the design and content of the measurement instruments and describes how measures generated by those instruments relate to the constructs. The ICILS 2023 Framework includes two domains of an individual’s skills with information communications technologies (ICT): computer and information literacy (CIL), which focuses on understanding computer use, gathering information, producing information, and communicating digitally, and computational thinking (CT), which focuses on conceptualizing problems and operationalizing solutions.
ICILS comprises both a student assessment and questionnaires for students, teachers, ICT coordinators, and school principals to gather information about the background contexts for ICT. The questionnaires are available here.
ICILS 2023 was conducted to help answer the following research questions, which apply to both the CIL and CT components:
- What variations in students’ CIL and/or CT skills exist within and across countries?
- How is CIL and/or CT education implemented across countries, and what school and country characteristics are related to students’ CIL and/or CT skills?
- How has CIL changed since ICILS 2013 and how has CT changed since ICILS 2018?
- What aspects of students’ personal and social backgrounds (such as their gender and socioeconomic background) are related to their CIL and/or CT skills?
- What are the relationships between students’ levels of access to, familiarity with, and self-reported proficiency in using computers and their CIL and/or CT skills?
- What is the association between students’ CIL and CT, and how has this changed since 2018?
Computer and information literacy (CIL)
is defined in the ICILS 2023 Assessment Framework
as “an individual’s ability to use computers to investigate,
create, and communicate in order to participate effectively at home, at school, in the workplace,
and in society.”
This definition relies on and brings together technical competence (computer literacy) and intellectual capacity (conventional literacies, including information literacy), both of which are necessary to navigate a digital environment.
CLOSE
The CIL proficiency levels were established in 2013 after consideration of the content and difficulty of the test items. The items and relative difficulty were analyzed to identify themes of content and process that could be used to characterize the different levels on the CIL achievement scale. This process was performed iteratively until each level showed distinctive characteristics, and the progression from low to high achievement across the levels was clear. The level boundaries—set at 407, 492, 576, and 661 scale points—form four proficiency levels. Student scores below 407 scale points indicate CIL proficiency below the lowest level targeted by the assessment instrument. The CIL proficiency levels did not change from 2013 to 2023.
The descriptions of each level are syntheses of the common elements of CIL knowledge, skills, and understanding described by the items within each level. They also describe the typical ways in which students working at a level demonstrate their proficiency. Each level of the scale articulates the characteristics of students’ engagement with computers for the purposes of accessing, using, and creating information, as well as for communicating with others.
For detailed descriptions of the CIL proficiency levels, see the ICILS 2023 international report.
CLOSE
CIL assessment modules include the following essential features:
- Students complete tasks solely on a computer.
- The tasks have a real-world cross-curricular focus.
- The tasks combine technical, receptive, productive, and evaluative skills.
- The tasks reference safe and ethical uses of computer-based information.
CIL test modules follow a linear narrative and comprise a set of questions and tasks based on real-world themes. Modules start with a series of smaller tasks that focus on skill execution and information management (each of which typically takes less than 1 minute to complete). These smaller tasks prepare the student to carry out a larger task (which typically takes 10–15 minutes to complete).
Following the ICILS 2023 assessment, the IEA released two CIL modules
(Breathing and School Trip). Visit the ICILS Released Assessment Questions page for
access to the modules.
CLOSE
Computational thinking (CT), first measured in ICILS 2018, is defined in the ICILS 2023 Assessment Framework as “an individual’s ability to recognize aspects of real-world problems which are appropriate for computational formulation and to evaluate and develop algorithmic solutions to those problems so that the solutions could be operationalized with a computer.” CT is the style of thinking used when programming a computer or developing an application for another type of digital device. However, its usefulness extends beyond this. The reasoning strategies that underlie CT can help make sense of complex ideas and solve problems. The assessment of CT through ICILS is an opportunity to gather international comparative data about how students are developing computational problem-solving skills at school.
CLOSE
For ICILS 2018, CT performance regions were created using a different method than the CIL proficiency levels. CT items were ordered by difficulty and then divided into three groups, with equal numbers of items in each group. The descriptions of each region were syntheses of the common elements of students’ CT knowledge, skills, and understanding described by the items within the region. There were three regions in the 2018 CT scale: lower (below 459 scale points), middle (between 459 and 589 scale points, inclusive), and upper (above 589 scale points). The regions of the CT scale should not be directly compared with the levels of the CIL scale.
For ICILS 2023, the CT performance regions were replaced with proficiency levels. The ICILS 2023 CT proficiency levels are not comparable with the ICILS 2018 CT preliminary regions. The items and relative difficulty were analyzed to identify themes of content and process that could be used to characterize the different levels on the CT achievement scale. This process was performed iteratively until each level showed distinctive characteristics, and the progression from low to high achievement across the levels was clear. The level boundaries—set at 330, 440, 550, and 660 scale points—form four proficiency levels. Student scores below 330 scale points indicate CT proficiency below the lowest level targeted by the assessment instrument.
The descriptions of each level are syntheses of the common elements of CT knowledge, skills, and understanding described by the items within each level. They also describe the typical ways in which students working at a level demonstrate their proficiency. Each level of the scale articulates the characteristics of students’ engagement with computers for conceptualizing problems and operationalizing solutions.
For detailed descriptions of the CT proficiency levels, see the ICILS 2023 international report.
CLOSE
CT test modules share the same essential features as CIL modules but have an additional focus on systems thinking and development of algorithmic solutions. Modules comprise a set of questions and tasks relating to real‐world problems that may be addressed with computer‐based solutions. The tasks assess a range of technical competencies, critical thinking, problem‐solving abilities, and evaluation skills. In addition, some tasks require students to create and execute block‐based algorithms, designed such that students can demonstrate aspects of computational and algorithmic thinking without the need to learn the syntax or features of a specific programming language.
The 2023 CT assessment contained four 25-minute test modules. In countries participating in the CT option, students completed two of the four CT test modules in randomized order after they had completed the CIL test and the student questionnaire.
Following the ICILS 2023 assessment, the IEA released two CT modules (Farm Drone and Automated Bus).
Visit the ICILS Released Assessment Questions page for access to the modules.
CLOSE