Interpreting NAEP Technology and Engineering Literacy Results

Overview of the Assessment
Reporting the Assessment Results—Scale Scores and Achievement Levels
Results Are Estimates
NAEP Reporting Groups
Exclusion Rates
Statistical Significance
Cautions in Interpretations

Overview of the Assessment

Nationally representative samples of schools and students participated in the 2018 NAEP technology and engineering literacy (TEL) assessment. The results of the TEL assessment are based on eighth-grade students from about 600 public and private schools across the nation. Approximately 15,400 eighth-graders were assessed in TEL.

Because of an increasing emphasis on technology and engineering skills inside and outside the classroom, in 2008 the National Assessment Governing Board set out to develop a framework for a national assessment of students’ knowledge and skills in technology and engineering. Completing this work involved the collaboration of technology and engineering experts, business leaders, educational policymakers, teachers, parents, and the general public who provided input via regional forums, webinars, and committee meetings to draft and refine the NAEP Technology and Engineering Literacy Framework.

Given the importance of technology and engineering literacy for all individuals in a world of increasingly rapid technological change, the Governing Board developed a framework for TEL that articulated the domain of technology and engineering knowledge and skills that are important for all students, not just those pursuing STEM-related careers. The TEL domain consists of three major interconnected content areas—Technology and Society, Design and Systems, and Information and Communication Technology—and three practices that cut across the content areas—Understanding Technological Principles, Developing Solutions and Achieving Goals, and Communicating and Collaborating. Read more about what the TEL assessment measures, how the TEL assessment was developed, who took the assessment, and how the assessment was administered.

Read more about what the TEL assessment measures, how the TEL assessment was developed, who took the assessment and how the assessment was administered.

Reporting the Assessment Results—Scale Scores and NAEP Achievement Levels

The results of student performance on the NAEP TEL assessment are presented in two ways: as average scores on the NAEP TEL scale and as the percentages of students attaining NAEP TEL achievement levels. The average scale scores represent how students performed on the assessment. The NAEP achievement levels represent how that performance measured up against set expectations for achievement. Thus, the average scale scores represent what students know and can do, while the NAEP achievement-level results indicate the degree to which student performance meets expectations of what they should know and be able to do. 

NAEP administered the TEL assessment beginning in 2014. For the first year of the TEL assessment in 2014, new scales and NAEP achievement levels were established. Scales for the three content areas and the three practices for grade 8 were developed separately—all scales range from 0–300 with a mean set at 150. NAEP achievement levels are reported as the percentages of students performing at or above three NAEP achievement levels (NAEP Basic, NAEP Proficient, and NAEP Advanced). 

Because the NAEP TEL scales were developed independently for each content area and practice, scale score results cannot be compared across content areas or practices. However, these reporting metrics greatly facilitate performance comparisons within each content area and practice from year to year and from one group of students to another in the same grade.

NAEP achievement-level results are presented in terms of NAEP TEL achievement levels as adopted by the National Assessment Governing Board, and are intended to measure how well students' actual achievement matches the achievement desired of them. For each grade tested, the Governing Board has adopted three NAEP achievement levels: NAEP Basic, NAEP Proficient, and NAEP Advanced. For reporting purposes, the NAEP achievement-level cut scores are placed on the TEL scales, resulting in four ranges: below NAEP Basic, NAEP Proficient, and NAEP Advanced. For reporting purposes, the NAEP achievement-level cut scores are placed on the TEL scales, resulting in four ranges: below NAEP Basic, NAEP Basic, NAEP Proficient, and NAEP Advanced.

NAEP achievement-level setting is based upon the TEL content framework and a standard-setting process involving a cross section of educators and interested citizens from across the nation who were asked to judge what students should know and be able to do relative to the content set out in the NAEP TEL framework. NAEP achievement-level setting is based on the collective judgments of a broadly representative panel of teachers, education specialists, and members of the general public. The authorizing legislation for the National Assessment of Educational Progress (NAEP) requires that the NAEP achievement levels be used on a trial basis until the Commissioner of the National Center for Education Statistics (NCES) determines that the NAEP achievement levels are reasonable, valid, and informative to the public (20 USC § 9622(e)(2)(C)). The NCES Commissioner's determination is to be based on a congressionally mandated, rigorous, and independent evaluation. The latest evaluation of the NAEP achievement levels was conducted by a committee convened by the National Academies of Sciences, Engineering and Medicine 2016. The evaluation concluded that further evidence should be gathered to determine whether the NAEP achievement levels are reasonable, valid, and informative. Accordingly, the NCES commissioner determined that the trial status of the NAEP achievement levels should be maintained at this time. In 2018 the National Assessment Governing Board issued a revised Policy Statement clarifying that the NAEP Proficient level is not intended to reflect grade-level performance expectations but is specific to performance on NAEP assessments. Read the Governing Board Policy Statement here.

Results Are Estimates

The average scores and percentages presented are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented are available through the NAEP Data Explorer.

Top

NAEP Reporting Groups

Results are provided for groups of students defined by shared characteristics—race or ethnicity, gender, eligibility for free/reduced-price school lunch, highest level of parental education, type of school, charter school, type of school location, region of the country, status students with disabilities, and status students identified as English learners. Based on participation rate criteria, results are reported for subpopulations only when sufficient numbers of students and adequate school representation are present. The minimum requirement is at least 62 students in a particular group from at least five primary sampling unit (PSUs). However, the data for all students, regardless of whether their group was reported separately, were included in computing overall results. Explanations of the reporting groups are presented below.

Race/Ethnicity

Prior to 2011, student race/ethnicity was obtained from school records and reported for the six mutually exclusive categories shown below:

• White
• Black
• Hispanic
• Asian/Pacific Islander
• American Indian/Alaska Native
  
• Other or unclassified

Students who identified with more than one of the other five categories were classified as “other” and were included as part of the "unclassified" category along with students who had a background other than the ones listed or whose race/ethnicity could not be determined.

In compliance with new standards from the U.S. Office of Management and Budget for collecting and reporting data on race/ethnicity, additional information was collected beginning in 2011 so that results could be reported separately for Asian students, Native Hawaiian/Other Pacific Islander students, and students identifying with two or more races. Beginning in 2011, all of the students participating in NAEP were identified by school reports as one of the seven racial/ethnic categories listed below:

• White
• Black or African American
• Hispanic
• Asian
• Native Hawaiian or other Pacific Islander
• American Indian or Alaska Native
• Two or More Races
  

Students identified as Hispanic were classified as Hispanic even if they were also identified with another racial/ethnic group. Students who identified with two or more of the other racial/ethnic groups (e.g., White and Black) would have been classified as “other” and reported as part of the "unclassified" category prior to 2011, but from 2011 on were classified as “Two or More Races.” Results for these students are presented under the "Two or More Races" category in the graphics and tables in the report.

When comparing the results for racial/ethnic groups from 2014 and 2018 to earlier assessment years, the 2014 and 2018 data for Asian and Native Hawaiian/Other Pacific Islander students were combined into a single Asian/Pacific Islander category. Information based on student self-reported race/ethnicity will continue to be reported in the NAEP Data Explorer.

Gender

Results are reported separately for male and female students.

Eligibility for Free/Reduced-Price School Lunch

As part of the Department of Agriculture's National School Lunch Program (NSLP), schools can receive cash subsidies and donated commodities in turn for offering free or reduced-price lunches to eligible children. NAEP first began collecting data in 1996 on student eligibility for NSLP as an indicator of poverty. Based on available school records, students were classified as either currently eligible for the free/reduced-price school lunch or not eligible. Eligibility for free and reduced-price lunches is determined by students' family income in relation to the federally established poverty level. Students whose family income is at or below 130 percent of the poverty level qualify to receive free lunch, and students whose family income is between 130 percent and 185 percent of the poverty level qualify to receive reduced-price lunch. For the period July 1, 2018 through June 30, 2019, for a family of four, 130 percent of the poverty level is $32,630 and 185 percent is $46,435. The classification applies only to the school year when the assessment was administered (i.e., the 2018–19 school year) and is not based on eligibility in previous years. If school records were not available, the student was classified as "Information not available." If the school did not participate in the program, all students in that school were classified as "Information not available." Because of the improved quality of the data on students' eligibility for NSLP, the percentage of students for whom information was not available has decreased compared to the percentages reported prior to the 2003 assessment. As a result of the passage of the Healthy, Hunger-Free Kids Act of 2010, schools can use a new universal meal service option, the "Community Eligibility Provision" (CEP). Through CEP, eligible schools can provide meal service to all students at no charge, regardless of economic status and without the need to collect eligibility data through household applications. CEP became available nationwide in the 2014-2015 school year; as a result, the percentage of students in many states categorized as eligible for NSLP may have increased in comparison to 2013 due to this provision. Therefore, readers should interpret NSLP trend results with caution. See the proportion of students in each category in the NAEP Data Explorer.

Parental Education

Parents' highest level of education is defined by the highest level reported by eighth-graders and twelfth-graders for either parent. Fourth-graders were not asked to indicate their parents' highest level of education because their responses in previous studies were highly variable, and a large percentage of them chose the "I don't know" option. Parental education attainment is one component used to measure student’s socioeconomic status (SES).

Type of School

The national results are based on a representative sample of students in both public schools and nonpublic schools. Nonpublic schools include private schools, Bureau of Indian Affairs schools, and Department of Defense schools. Private schools include Catholic, Conservative Christian, Lutheran, and other private schools. Results are reported for private schools overall, as well as disaggregated by Catholic and other private schools. The school participation rates for private schools overall in 2014 met the 70 percent criteria for reporting at grade 8 but did not meet the NAEP reporting standards in 2018; therefore, their results are reported in the “type of school” charts on the website in 2014 but not in 2018. The school participation rate for Catholic schools did not meet the reporting standards in 2014 at grade 8 but met the standards in 2018; therefore, their results are not available in 2014 but are presented in the report in 2018.

Charter School

A pilot study of America's charter schools and their students was conducted as part of the 2003 NAEP assessments in reading and mathematics at grade 4. Results are available for charter schools starting in 2003 at grade 4, 2005 at grade 8, and 2009 at grade 12.

Type of Location

NAEP results are reported for four mutually exclusive categories of school location: city, suburb, town, and rural. The categories are based on standard definitions established by the Federal Office of Management and Budget using population and geographic information from the U.S. Census Bureau. Schools are assigned to these categories in the NCES Common Core of Data based on their physical address. In 2007, the classification system was revised; therefore, results for school location are available for 2007 and later assessment years.

In 2007, the classification system was revised; therefore, trend comparisons to previous years are not available. The new locale codes are based on an address's proximity to an urbanized area (a densely settled core with densely settled surrounding areas). This is a change from the original system based on metropolitan statistical areas. To distinguish the two systems, the new system is referred to as "urban-centric locale codes." The urban-centric locale code system classifies territory into four major types: city, suburban, town, and rural. Each type has three subcategories. For city and suburb, these are gradations of size—large, midsize, and small. Towns and rural areas are further distinguished by their distance from an urbanized area. They can be characterized as fringe, distant, or remote

Region

Prior to 2003, NAEP results were reported for four NAEP-defined regions of the nation: Northeast, Southeast, Central, and West. As of 2003, to align NAEP with other federal data collections, NAEP analysis and reports have used the U.S. Census Bureau's definition of "region." The four regions defined by the U.S. Census Bureau are Northeast, South, Midwest, and West. The Central region used by NAEP before 2003 contained the same states as the Midwest region defined by the U.S. Census. The former Southeast region consisted of the states in the Census-defined South minus Delaware, the District of Columbia, Maryland, Oklahoma, Texas, and the section of Virginia in the District of Columbia metropolitan area. The former West region consisted of Oklahoma, Texas, and the states in the Census-defined West. The former Northeast region consisted of the states in the Census-defined Northeast plus Delaware, the District of Columbia, Maryland, and the section of Virginia in the District of Columbia metropolitan area. The table below shows how states are subdivided into these Census regions. All 50 states and the District of Columbia are listed. Other jurisdictions, including the Department of Defense Educational Activity schools, are not assigned to any region.

States within regions of the country defined by the U.S. Census Bureau

Northeast	South	Midwest	West
Connecticut Maine Massachusetts New Hampshire New Jersey New York Pennsylvania Rhode Island Vermont	Alabama Arkansas Delaware District of Columbia Florida Georgia Kentucky Louisiana Maryland Mississippi North Carolina Oklahoma South Carolina Tennessee Texas Virginia West Virginia	Illinois Indiana Iowa Kansas Michigan Minnesota Missouri Nebraska North Dakota Ohio South Dakota Wisconsin	Alaska Arizona California Colorado Hawaii Idaho Montana Nevada New Mexico Oregon Utah Washington Wyoming

Students with Disabilities (SD)

Results are reported for students who were identified by school records as having a disability. A student with a disability may need specially designed instruction to meet his or her learning goals. A student with a disability will usually have an Individualized Education Program (IEP) which guides his or her special education instruction. Students with disabilities (SD) are often referred to as special education students and may be classified by their school as learning disabled (LD) or emotionally disturbed (ED). The goal of NAEP is that students who are capable of participating meaningfully in the assessment are assessed, but some students with disabilities selected by NAEP may not be able to participate, even with the accommodations provided. Beginning in 2009, NAEP disaggregated students with disabilities from students who were identified under section 504 of the Rehabilitation Act of 1973; however, trend results dating back to 1998 are available in reading for the SD variable that includes section 504 students. The results for SD are based on students who were assessed and could not be generalized to the total population of such students.

English Learners (EL)

Results are reported for students who were identified by school records as being English learners (EL). (Note that English learners were previously referred to as limited English proficient (LEP). The results for EL are based on students who were assessed and could not be generalized to the total population of such students.

Top

Exclusion Rates

Assessing representative samples of students, including students with disabilities (SD) and English learners (EL) , helps to ensure that NAEP results accurately reflect the educational performance of all students in the target population, and can continue to serve as a meaningful measure of U.S. students’ academic achievement over time. See tables that summarize the percentage of students identified, excluded, and assessed in TEL.

To ensure that all selected students from the population can be assessed, many of the same accommodations that SD and EL students use on other tests are provided for those students participating in NAEP. NAEP’s accommodation practices are evolving along with digitally based assessments (DBA). Instead of using a “one-size-fits-all” approach, DBA allows NAEP assessments to be more flexible in meeting the needs of different students. Read more about accommodations available in NAEP.

Statistical Significance

The differences between scale scores and between percentages discussed in the results take into account the standard errors associated with the estimates. Comparisons are based on statistical tests that consider both the magnitude of the difference between the group average scores or percentages and the standard errors of those statistics. Throughout the results, differences between scores or between percentages are discussed only when they are significant from a statistical perspective.

All differences reported are significant at the 0.05 level with appropriate adjustments for multiple comparisons. The term "significant" is not intended to imply a judgment about the absolute magnitude or the educational relevance of the differences. It is intended to identify statistically dependable population differences to help inform dialogue among policymakers, educators, and the public.

Cautions in Interpretations

Users are cautioned against interpreting NAEP results as implying causal relations. Inferences related to group performance or to the effectiveness of public and nonpublic schools, for example, should take into consideration the many socioeconomic and educational factors that may also impact performance.

The NAEP TEL scale makes it possible to examine relationships between students' performance and various factors measured by NAEP. However, a relationship that exists between achievement and another variable does not reveal its underlying cause, which may be influenced by a number of other variables. Similarly, the assessments do not reflect the influence of unmeasured variables. The results are most useful when they are considered in combination with other knowledge about the student population and the educational system, such as trends in instruction, changes in the school-age population, and societal demands and expectations.