In addition to assessing subject area achievement, NAEP collects information from participating students, teachers, and schools about background variables that are related to student achievement. This information serves, in part, to fulfill reporting requirements of federal legislation. Specifically, under the No Child Left Behind Act, NAEP is required to collect information on and report achievement results disaggregated by the following variables, when possible: gender, race and ethnicity, socioeconomic status (SES), disability status, and English language learner (ELL) status. (Note that the term English language learner is used in NAEP 2005 reports; the term limited English proficient was used before 2005 and was used on all SD/ELL questionnaires administered to schools up to and including 2005.) Information from the background items also serves to give context to NAEP results and/or allow researchers to track factors associated with academic achievement.
In early 2002, the National Assessment Governing Board was granted final authority over the background items. The Board adopted a policy to focus NAEP background data on the primary purpose of the National Assessment—to provide sound, timely information on the academic achievement of students in the United States (National Assessment Governing Board, 2003). The Board also initiated a process to prepare a general framework to guide the collection and reporting of background data. The Background Information Framework for the National Assessment of Educational Progress (NAEP), developed in 2003, defines the purpose and scope of NAEP background data, and calls for a long-term plan for continued development. In response to this call, the National Center for Education Statistics (NCES) developed the NCES Plan for NAEP Background Variable Development, which provides a general procedural map for the development and review of each type of background data (National Center for Education Statistics, 2004).
There are three types of background data: student reporting categories, other contextual/policy information, and subject-specific information. While there are some differences in the approaches to the development of each type of data, shared principles underlie all three: The Governing Board provides initial guidance on what will be developed; the Board has multiple opportunities to review and provide input; and the overall development process seeks to reduce burden on respondents and ensure data quality while continuing to meet the needs of the NAEP program. Descriptions of the three types of background data are below.
General Student Reporting Categories
Since the first NAEP assessment in 1969, achievement results have been disaggregated by subgroups of the population. Achievement has also been presented for and compared across subgroups. As mentioned earlier, since the inception of the No Child Left Behind Act, NAEP has collected information on and reported achievement results disaggregated by the following variables: gender, race and ethnicity, socioeconomic status, disability status, and English language learner status.
NCES monitors the quality of the data collected using the current measures and will develop new approaches to measuring student reporting variables when warranted.
One new approach NCES is currently investigating is the creation of a new and improved measure of socioeconomic status (SES). NCES commissioned a literature review of how SES has been defined and operationalized in other education studies and other fields such as health and marketing. This led to the idea of adopting a two-pronged approach to measuring SES. This approach involves (1) creating an enhanced student background questionnaire with items that probe resources in the home, parents’ education level, and parents’ employment status, among other variables; and (2) using geocoding software to link students’ home addresses to aggregate SES data available from the United States Bureau of the Census. Development of the new SES measure commenced in 2005, with the goal of piloting it in 2009 and possibly implementing it in 2011.
In every assessment, NAEP collects data on basic characteristics of the school and student body in the school; teacher background, qualifications, and experience; and several student characteristics. These variables provide a basic context for achievement.
In addition to these core variables, timely policy/contextual issues are rotated across assessments. NCES convenes a policy/contextual issues panel when needed to identify policy/contextual issues that NAEP might address in the future, and to outline the relevant constructs and identify data needed to address these issues.
The subject-specific items in NAEP are focused and limited. A set of key issues within each subject area will be addressed in a focused and in-depth manner across the life of each assessment framework.
When a new assessment framework is approved, NCES reviews the recommendations for background data made by the framework committee. Since 2003, NCES then develops an issues paper to reflect those priorities, identifies the data needed to address the issues, and develops a proposed schedule for rotating topics.
Background items associated with the categories described above are placed within student, teacher, school, and/or SD/ELL questionnaires, as appropriate. The placement of items and content of each questionnaire depend on the questionnaire respondent and the specific subject(s) NAEP is assessing in a given year. Often questionnaires measure similar constructs across respondents and/or subjects to provide additional information and, in some cases, to validate findings.
Student background questionnaires ask respondents to provide information about factors such as race or ethnicity, school attendance, and academic expectations. Responses to items on the questionnaires also provide information about factors associated with academic performance, students’ educational settings and experiences, students’ effort on the assessment, and the difficulty and importance of the assessment.
Teacher questionnaires ask respondents to indicate teacher background, training, and instructional practices (completed by teachers at grades 4 and 8. NAEP typically does not collect teacher information at grade 12.)
School questionnaires ask respondents to provide information on school policies and characteristics (completed by the principal or assistant principal.)
Questionnaires about students with disabilities or English language learners questionnaires ask respondents to provide information about students selected in the sample who have disabilities or limited English proficiency (completed by a special education teacher, a bilingual education/English-language-learner teacher, or a staff member who is most familiar with the student.)
In 2005 NCES also piloted a department head questionnaire for grade 12 economics. Within each participating school, the questionnaire was administered to the chair or lead teacher of every department that offered at least one economics-related course. The questionnaire asked the respondent to provide information about the characteristics of the department’s faculty, hiring requirements, and courses offered by the department. There are currently no plans to administer the department head questionnaire in every NAEP assessment. For more information about these questionnaires and their content, refer to the links to the right or to Instruments for the National Assessment of Educational Progress (NAEP).
Background items are developed through a process similar to that used for developing the cognitive items. It includes reviews by external advisory groups and field testing. When developing the items, NAEP ensures that the items do not infringe on respondents' privacy, that they are grounded in educational research, and that the answers can provide information relevant to the subject being assessed. The following is an overview of the development process for background items:
1. The National Assessment Governing Board oversees the development of the content framework and item specifications for the background items. More details about this process are provided in the Background Information Framework (NAGB, 2003).
2. When a new assessment framework is approved, or when new policy issues are identified for NAEP to address, NCES develops an issues paper to reflect the new priorities, identify the data needed to address the issues, and propose an item rotation plan. The development of the issues paper involves convening a panel of experts in the relevant fields to help identify issues and then conducting a literature review to identify recent developments for the respective issues.
3. NAEP contractors that specialize in survey development draft and revise background items based on the recommendations of the issues paper and expert panel. Again, issue-specific working groups and expert panels are convened to provide input on the items.
4. NCES then reviews the background items to ensure fairness and quality so that NAEP’s mission of providing a fair and accurate measure of student achievement and achievement trends over time is fulfilled (see NCES Statistical Standards).
5. The items are piloted, and the results are analyzed.
6. Based upon pilot data results, some items are revised.
7. The background items once again undergo reviews by item development contractors and then by NCES.
8. NCES presents items to the Governing Board for its approval, as specified in Education Sciences Reform Act, P.L. 107-279 . The Board has "final authority on the appropriateness of all assessment items" and is required "to take steps to ensure that all items selected for use in the National Assessment are free from racial, cultural, gender, or regional bias and are secular, neutral, and non-ideological."
9. The items are then submitted for clearance by NCES to the Office of Management and Budget, which checks to make sure the items comply with government policies.
10. Once clearance is received, each background item is typeset into the respective student, teacher, school, and/or SD/ELL questionnaires.
The purpose of administering background items is to give context to NAEP results and/or to track factors associated with academic achievement. The data are also the basis for NAEP’s major reporting groups. Therefore, it is important to note that since NAEP is based on a cross-sectional design, it is not possible to infer cause-and-effect relationships—it cannot prescribe what should be done. Rather its descriptions of the educational circumstances of students at various achievement levels—considered in light of research from other sources—may provide important information for public discussion and policy action (NAGB, 2003). For more information regarding how NAEP data is analyzed and reported refer to the “Results” section of NAEP’s Frequently Asked Questions or the Background Information Framework for the National Assessment of Educational Progress (NAEP) developed by the Governing Board. For more information on how you can explore and manipulate NAEP data, go to the NAEP Research e-Center or the NAEP Data Explorer. Please note that in the NAEP Data Explorer, the results of the background questionnaires are sorted into eight broad categories: