​​​​NAEP Technical DocumentationContextual Items (Survey Questionnaires)

			Student Questionnaire Teacher Questionnaire School Questionnaire SD/EL Questionnaire Department Head Questionnaire Indices Based on Questionnaires

In addition to assessing subject area achievement, NAEP collects information from participating students, teachers, and schools through survey questionnaires that measure contextual factors related to student achievement. This information serves to fulfill reporting requirements of federal legislation in conjunction with school records. NAEP is required to collect information on and report achievement results disaggregated by the following variables, when possible: gender, race/ethnicity, socioeconomic status (SES), disability status (SD), and English learner (EL) status. (Note that the term English learner is used in NAEP reports on the assessments administered since and including 2019; the term limited English proficient [LEP] was used in SD/EL questionnaires administered to schools up to and including 2005, and the term English language learner [ELL] was used from 2005 through 2018.) NAEP also strives to conform with the National Assessment Governing Board's 2012 policy statement¹ that "NAEP reporting should be enriched by greater use of contextual data derived from background or non-cognitive questions asked of students, teachers, and schools" (National Assessment Governing Board 2012, p. 2) by collecting information and reporting achievement results for additional topics, such as technology use, school climate, and students' persistence in learning.

Information from the survey questionnaires also serves to provide context for reporting and interpreting NAEP results and allows researchers to examine factors associated with academic achievement.

Contextual Data

The purpose of administering survey questionnaires is to give context to NAEP results and to examine factors associated with academic achievement. The data are also the basis for identifying NAEP’s major reporting groups in conjunction with school records. It is important to note that since NAEP is based on a cross-sectional design, it is not possible to infer cause-and-effect relationships—it cannot prescribe what should be done. Rather, its descriptions of the educational circumstances of students at various achievement levels—considered in light of research from other sources—may provide important information for public discussion and policy action (National Assessment Governing Board 2003). 

For more information regarding how NAEP data are reported, refer to NAEP’s Frequently Asked Questions or the Background Information Framework for the National Assessment of Educational Progress (NAEP) in the National Assessment Governing Board's 2012 policy statement. NAEP reports can be accessed on The Nation's Report Card website, and the data can be accessed by the general public via the NAEP Data Explorer (NDE). NAEP does not report on individual student or school responses. In addition to the national results and data which are available for all subjects and student groups, state-level NAEP results and data for selected urban districts can be reported for some subjects and student groups. 

Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP tests do not make information about test takers' personal identity publicly available. To further ensure privacy and anonymity, names are not included in any data files. All participants in the NAEP assessment are provided with unique individual identification numbers to ensure that their personal information cannot be linked directly to answers they provide. 

For more information on how you can explore and manipulate NAEP data, go to the NAEP Research e-Center or the NDE. Please note that there is some variation in the survey questions administered to students across years and subjects, so the contextual variables that are available in the NDE may likewise differ from year to year and subject to subject. Within the NDE, the results of the survey questionnaires for any given year and subject may include variables that are organized into the following broad categories:

• major reporting groups;
  
• student factors;
  
• instructional content and practice;
  
• teacher factors;
  
• school factors;
  
• community factors;
  
• factors beyond school;
  
• government factors; and 
  
• achievement level as a variable.
  

Types of Survey Questions

There are three types of contextual information collected through the survey questionnaires: general student reporting categories, other contextual/policy information, and subject-specific information. While there are some differences in the approaches to the development of each type of data, shared principles underlie all three:  

• All NAEP survey questions undergo a multi-step research-based development process.
• Multiple groups review the NAEP survey questions and provide input at each stage of the process, including survey experts, subject-area experts, educational researchers, statisticians, and the National Assessment Governing Board (the Governing Board).  
  
• The overall development process seeks to reduce the burden on respondents and ensure data quality while continuing to meet the needs of the NAEP program.

Descriptions of the three types of contextual information are below.

• General Student Reporting Categories

  • Since the first NAEP assessment in 1969, achievement results have been disaggregated by selected student groups in the population. Achievement has also been presented for and compared across these groups. NAEP collects information on and reports achievement results disaggregated by the following contextual variables: gender, race/ethnicity, socioeconomic status, disability status, and English learner status.

  • NCES monitors the quality of the data collected using the current measures (i.e., school records and survey questionnaires) and will develop new approaches (e.g., collecting data or using measures from other NAEP questionnaires and studies) to measuring contextual variables when warranted.
    

• Other Contextual/Policy Information

  • In every assessment, NAEP collects data on basic contextual information about the school and student body in the school; teacher background, qualifications, and experience; and several student characteristics. These variables provide a basic context for understanding achievement.

• Subject-Specific Information

  • The subject-specific survey questions in NAEP provide focused and in-depth information to address key issues identified within each subject area. These issues are addressed across the life of each assessment framework.

  • When a new assessment framework is approved, NCES reviews the recommendations for contextual information made by the framework committee. NCES then develops an issues paper to reflect those priorities, and identifies the data needed to address the issues.

Survey questions associated with the categories described above are placed within student, teacher, school, and SD/EL questionnaires, as appropriate. The placement of questions and content of each survey questionnaire depend on the respondent and the specific subject(s) assessed in a given year. Often the survey questionnaires include several questions on the same topic to enable the creation of contextual indices that measure a single underlying construct or concept with greater validity. This approach entails developing multiple survey questions associated with the same construct and examining the relationship of the questions with each other and with the construct (i.e., while each item uniquely contributes to measuring the construct, they should still holistically measure the same topic). 

• Student questionnaires collect information about factors such as students' demographic characteristics, opportunities to learn in and outside of the classroom, and academic experiences (completed by students). 

• Teacher questionnaires gather information on teacher training and instructional practices (completed by teachers of the students being assessed at grades 4 and 8; NAEP typically does not collect teacher information at grade 12²).

• School questionnaires gather information on school policies and characteristics (completed by the principal or assistant principal). To ensure that necessary accommodations are provided, schools also provide information about students selected in the sample who have disabilities (SD) or are English learners (EL). There is also a supplemental charter school questionnaire designed to collect information on charter school policies and characteristics. Principals (or other administrators) of charter school students sampled to participate in NAEP complete both the school questionnaire and the charter school supplement. 

Survey Question Development Process

Survey questions are developed through a process similar to that used for developing the cognitive items. This process includes extensive reviews by external advisory groups, cognitive interviews, and pilot testing. When developing the questions, NAEP ensures that the questions are grounded in educational research, and that the answers can provide information relevant to the subject being assessed. The following is an overview of the development process for contextual questions:

1. The framework for survey questionnaires is one resource that guides the development process and the factors of interest to be measured via the survey questions (National Center for Education Statistics 2004).
2. When a new assessment framework is approved, or when new policy issues are identified for NAEP to address, NCES conducts a literature review and develops an issues paper to reflect the new priorities, and identifies the data needed to address the issues. 
3. NAEP contractors specializing in survey development draft and revise survey questions based on the recommendations of the issues paper and a panel of experts.
4. NCES reviews the survey questions to ensure they are fair, relevant, and useful for stakeholders to contextualize student achievement and achievement trends over time (see the Contextual Information Framework for the National Assessment of Educational Progress).
   
5. New and revised questions undergo cognitive interview testing, in which participants are interviewed to identify potential issues with their comprehension of the survey questions and their ability to provide reliable and valid answers. Based on the results of the cognitive interviews, some questions may be dropped or further revised.
6. The questions are piloted, the results are analyzed, and questions are revised if needed.
7. The survey questions once again undergo reviews by item development contractors and NCES.
   
8. NCES presents the survey questions to the Governing Board before a pilot or operational administration, as specified in Education Sciences Reform Act, P.L. 107-279. The Board serves as a critical external reviewer on the appropriateness of the assessment items, "to ensure that all items selected for use in the National Assessment are free from racial, cultural, gender, or regional bias and are secular, neutral, and non-ideological."
9. Prior to administration of each cognitive interview, pilot testing, and operational assessment, the survey questions are submitted for clearance by NCES to the Office of Management and Budget, which checks to make sure the questions comply with government policies.
10. After pilot or operational administration clearance is received, PDF and online versions of the student, teacher, and school questionnaires are finalized.
    

Printer-friendly Version