Skip to main content
Skip Navigation

Table of Contents | Search Technical Documentation | References

NAEP Instruments → Contextual Items (Survey Questionnaires)

NAEP Technical DocumentationContextual Items (Survey Questionnaires)

      

Student Questionnaire

Teacher Questionnaire

School Questionnaire

SD/EL Questionnaire

Department Head Questionnaire

Indices Based on Questionnaires

           

Established in 1969, the National Assessment of Educational Progress (NAEP), also known as "The Nation's Report Card," serves as a common yardstick for measuring the progress of students' education across the country over time. In addition to assessing subject area achievement, NAEP collects information from participating students, teachers, and schools through survey questionnaires that measure contextual factors related to student achievement. 

This information serves to fulfill reporting requirements of federal legislation in conjunction with school records. NAEP is required to collect information on and report achievement results disaggregated by the following variables, when possible: gender, race and ethnicity, socioeconomic status (SES), disability status (SD), and English learner (EL) status. (Note that the term English language learner is used in NAEP reports administered since and including 2005; the term limited English proficient [LEP] was used in SD/EL questionnaires administered to schools up to and including 2005.) 

Information from the survey questionnaires also serves to provide a context for reporting and interpreting NAEP results and allows researchers to examine factors associated with academic achievement.

Recent History

In line with the National Assessment Governing Board's 2012 policy statement that "NAEP reporting should be enriched by greater use of contextual data derived from background or non-cognitive questions asked of students, teachers, and schools" (p. 2), NAEP survey questionnaires are currently being enhanced through the development of new survey questions clustered around five broad topics: socioeconomic status, technology use, school climate, grit, and desire for learning. Questions for these new topics have been tested with small samples of participants and were piloted in a nationally representative sample in early 2016. Based on data from the pilot, a final set of survey questions for the 2017 NAEP administration was selected. 

Types of Survey Questions

There are three types of contextual information collected through the survey questionnaires: general student reporting categories, other contextual/policy information, and subject-specific information. While there are some differences in the approaches to the development of each type of data, shared principles underlie all three:  

  • All NAEP survey questions undergo a multi-step research-based development process.
  • Multiple groups review the NAEP survey questions and provide input at each stage of the process, including survey experts, subject-area experts, educational researchers, statisticians, and the National Assessment Governing Board (the Governing Board).  
  • The overall development process seeks to reduce the burden on respondents and ensure data quality while continuing to meet the needs of the NAEP program.

Descriptions of the three types of contextual information are below.

  • General Student Reporting Categories

    • Since the first NAEP assessment in 1969, achievement results have been disaggregated by selected student groups in the population. Achievement has also been presented for and compared across these groups. NAEP collects information on and reports achievement results disaggregated by the following contextual variables: gender, race and ethnicity, socioeconomic status, disability status, and English learner status.

    • NCES monitors the quality of the data collected using the current measures (i.e., school records and survey questionnaires) and will develop new approaches (e.g., collecting data or using measures from other NAEP questionnaires and studies) to measuring contextual variables when warranted.

  • Other Contextual/Policy Information

    • In every assessment, NAEP collects data on basic contextual information about the school and student body in the school; teacher background, qualifications, and experience; and several student characteristics. These variables provide a basic context for understanding achievement.

  • Subject-Specific Information

    • The subject-specific survey questions in NAEP provide focused and in-depth information to address key issues identified within each subject area. These issues are addressed across the life of each assessment framework.

    • When a new assessment framework is approved, NCES reviews the recommendations for contextual information made by the framework committee. NCES then develops an issues paper to reflect those priorities, and identifies the data needed to address the issues.

Survey questions associated with the categories described above are placed within student, teacher, school, and/or SD/EL questionnaires, as appropriate. The placement of questions and content of each survey questionnaire depend on the respondent and the specific subject(s) NAEP is assessing in a given year. Often the survey questionnaires include multiple measures of a construct to enable the development of contextual indices with greater validity for the desired construct. This approach entails having multiple survey questions associated with the same construct and examining the relationship of the questions with each other and with the construct (i.e., while each item uniquely contributes to measuring the construct, they should still holistically measure the same topic). 

  • Student questionnaires collect information about factors such as students' demographic characteristics, opportunities to learn in and outside of the classroom, and academic experiences (completed by students). 

  • Teacher questionnaires gather information on teacher training and instructional practices (completed by teachers of the subjects being assessed at grades 4 and 8; NAEP typically does not collect teacher information at grade 121).

  • School questionnaires gather information on school policies and characteristics (completed by the principal or assistant principal). To ensure that necessary accommodations are provided, schools also provide information about students selected in the sample who have disabilities (SD) or limited English proficiency (EL). There is also a supplemental charter school questionnaire designed to collect information on charter school policies and characteristics. Principals (or other administrators) of charter school students sampled to participate in NAEP complete both the school questionnaire and the charter school supplement. 

Survey Question Development Process

Survey questions are developed through a process similar to that used for developing the cognitive items. This process includes extensive reviews by external advisory groups, cognitive interviews, and pilot testing. When developing the questions, NAEP ensures that the questions are grounded in educational research, and that the answers can provide information relevant to the subject being assessed. The following is an overview of the development process for contextual questions:

  1. The framework for survey questionnaires is one resource that guides the development process and the factors of interest to be measured via the survey questions (National Center for Education Statistics 2004).
  2. When a new assessment framework is approved, or when new policy issues are identified for NAEP to address, NCES conducts a literature review and develops an issues paper to reflect the new priorities, and identifies the data needed to address the issues. 
  3. NAEP contractors specializing in survey development draft and revise survey questions based on the recommendations of the issues paper and a panel of experts.
  4. NCES reviews the survey questions to ensure they are fair, relevant, and useful for stakeholders to contextualize student achievement and achievement trends over time (see the Contextual Information Framework for the National Assessment of Educational Progress).
  5. New and revised questions undergo cognitive interview testing, in which participants are interviewed to identify potential issues with their comprehension of the survey questions and their ability to provide reliable and valid answers. Based on the results of the cognitive interviews, some questions may be dropped or further revised.
  6. The questions are piloted, the results are analyzed, and questions are revised if needed.
  7. The survey questions once again undergo reviews by item development contractors and NCES.
  8. NCES presents the survey questions to the Governing Board before a pilot or operational administration, as specified in Education Sciences Reform Act, P.L. 107-279. The Board serves as a critical external reviewer on the appropriateness of the assessment items, "to ensure that all items selected for use in the National Assessment are free from racial, cultural, gender, or regional bias and are secular, neutral, and non-ideological."
  9. Prior to cognitive interviews, pilot testing, and operational administration, the survey questions are submitted for clearance by NCES to the Office of Management and Budget, which checks to make sure the questions comply with government policies.
  10. After pilot or operational administration clearance is received, PDF and online versions of the student, teacher, and school questionnaires are finalized.

Contextual Data

The purpose of administering survey questionnaires is to give context to NAEP results and/or to examine factors associated with academic achievement. The data are also the basis for NAEP’s major reporting groups in conjunction with school records. It is important to note that since NAEP is based on a cross-sectional design, it is not possible to infer cause-and-effect relationships—it cannot prescribe what should be done. Rather, its descriptions of the educational circumstances of students at various achievement levels—considered in light of research from other sources—may provide important information for public discussion and policy action (National Assessment Governing Board 2003). 

For more information regarding how NAEP data are reported, refer to NAEP’s Frequently Asked Questions or the Background Information Framework for the National Assessment of Educational Progress (NAEP) developed by the Governing Board. NAEP reports can be accessed on The Nation's Report Card website, and the data can be accessed by the general public via the NAEP Data Explorer (NDE). NAEP does not report on individual student or school responses. In addition to the national results and data which are available for all subjects and student groups, state-level NAEP results and data for selected urban districts can be reported for some subjects and student groups. 

Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP tests do not make information about test takers' personal identity publicly available. To further ensure privacy and anonymity, names are not included in any data files. All participants in the NAEP assessment are provided with unique individual identification numbers to ensure that their personal information cannot be linked directly to answers they provide. 

For more information on how you can explore and manipulate NAEP data, go to the NAEP Research e-Center or the NAEP Data Explorer (Data Explorer). Please note that there is some variation in the survey questions administered to students across years and subjects, so the contextual variables that are available in the Data Explorer will likewise differ from year to year and subject to subject. Within the Data Explorer, the results of the survey questionnaires for any given year and subject may include variables that are organized into the following broad categories:

  • major reporting groups,
  • student factors,
  • instructional content and practice,
  • teacher factors,
  • school factors,
  • community factors,
  • factors beyond school,
  • government factors, and 
  • achievement level as a variable.
footnote1In 2006 and 2012, NCES also administered a department head questionnaire for grade 12 economics. Within each participating school, the survey questionnaire was administered to the chair or lead teacher of every department that offered at least one economics-related course. The survey questionnaire asked the respondent to provide information about the characteristics of the department's faculty, hiring requirements, and courses offered by the department. There are currently no plans to administer the department head questionnaire in other NAEP assessments. 

 


Last updated 16 August 2022 (SK)