Vol. 1, No. 2
October 1994
(NCES 94-666) Ordering Information
In spring 1994, the National Assessment of Educational Progress (NAEP) collected information on student achievement in reading, U.S. history, and geography at grades 4, 8, and 12. To provide a context for interpreting NAEP findings, the project also collected background information from students, teachers, and school administrators, using separate questionnaires.
This Focus on NAEP describes the NAEP teacher background questionnaire: How were the questions developed? What are the types of questions? How are they used?
Teachers selected to answer background questions are those who teach classes containing one or more students participating in the assessment. Because the sampling for the teacher questionnaire is based on participating students, the responses do not represent all teachers in the nation or a state. Rather, the sample consists of teachers of the representative sample of students assessed.
The teacher questionnaire is given to teachers of assessed students at grades 4 and 8 only. At grade 12, relatively few students take the assessed subject areas of reading, U.S. history, and geography; therefore, the teacher sample sizes are insufficiently large to give the questionnaire. The questionnaire has two sections. The first focuses on teachers' background, education, resources, and preparation for teaching in specific subject areas. The second focuses on classroom information including instructional content and practices related to each subject area.
NAEP's teacher background questions (along with the background questions for students and school administrators) are developed through a broad-based consensus process. For the 1994 assessment, information was gathered about appropriate background
variables from many sources. NAEP's governing board convened national consensus committees of teachers, scholars, business representatives, policymakers, measurement specialists, and members of the public to recommend background variables for study. Once developed, this information, together with the 1992 background questionnaires, was submitted for mail review to 30 education professionals.
A 10-member Background Questionnaire Committee, including a member of each subject area committee and 7 education professionals who had participated in the mail review, was convened by Educational Testing Service, the NAEP grantee, to help develop and revise the questionnaires for the 1994 assessment. Subject-specific questions were developed and reviewed by members of the Instrument Development Committee in all three subject areas. Finally, state representatives
belonging to the NAEP Trial State Assessment NETWORK reviewed them. After they were reviewed and cleared by the Office of Management and Budget, they were field tested in actual classrooms in 1993 under conditions similar to those in the actual 1994 assessment.
The individuals and committees who developed the 1994 background questions were instructed to include only factors identified by NAEP and other studies to be important to student learning and of current interest to policymakers, researchers, and content-area educators. To maintain NAEP's ability to report trends over time, many NAEP teacher background questions were drawn from prior assessments.
In sum, the teacher questionnaire is the product of extensive reviews and deliberations by many groups of individuals, with expertise in classroom teaching, survey methodology, policy analysis, and educational research.
Table 1 is a matrix for conceptualizing the type, content, and uses of the teacher background questions. The first dimension specifies three analysis issues that background questions address including instructional content, instructional practices, and teacher background, education, and resources. The second dimension specifies the purposes of the questions: to describe existing educational conditions, instructional factors by demographic subgroup, trends in instruction, relation of instruction to performance, and changes in performance in relation to trends in instruction. The intersecting cells of the matrix provide examples of research questions that may be examined under this conceptual framework.
The three types of background questions-instructional content, instructional practices, and teacher background education, and resources-are broad and inclusive enough to address the needs, interests, and concerns of NAEP audiences. Knowing what content and instructional methods students have been exposed to in the classroom is essential to interpreting the assessment results, particularly when comparing student groups.
---------------------------------------------------------------------------- Table 1.-The NAEP teacher background questionnaire: Analysis issues and purposes ---------------------------------------------------------------------------- Analysis Issues ---------------------------------------------------------------------------- Purpose Instructional Instructional content practices Existing What are teachers How are students educational teaching? taught? conditions Instructional How does instructional How do instructional factors by content vary for practices vary for demographic different groups different groups subgroup of students? of students? Trends in How has content Have instructional instruction changed over time? practices changed over time? Relation of Do students exposed Do students exposed instruction to to particular material to particular practices performance score higher? score higher? Changes in Does performance Does performance performance in change in relation change in relation to relation to trends to changes in to changes in in instruction instructional content? instructional practices? ---------------------------------------------------------------------------- Table 1.-Continued ---------------------------------------------------------------------------- Analysis issues ---------------------------------------------------------------------------- Purpose Teacher background education, and resources Existing What kinds of educational educational and background experiences conditions do teachers bring to the classroom? Instructional Are specific groups of factors by students taught by teachers demographic with different backgrounds? subgroup Trends in Have teacher characteristics instruction changed over time? Relation of Does relevant training instruction to and experience relate to performance better teaching? Changes in Does performance performance in change in relation to relation to trends changes in teacher in instruction characteristics? ----------------------------------------------------------------------------
Instructional Content. Opportunities to learn are affected by the content covered in the classroom. The curriculum may also be implemented differently by individual teachers in their classes. As a result, the extent to which topics, concepts, and skills are covered may vary from teacher to teacher and class to class. Information about students' exposure to reading, U.S. history, and geography will answer many questions such as the following:
---------------------------------------------------------------------------- Sample Instructional Content Questions What type of materials form the core of your reading programs? (primarily basal, primary trade books, etc.) [reading] Which best describes the focus of your social studies/history teaching? (state history, etc) [U.S. history] How often do you teach the following skills and topics as part of geography by instruction with this class? (measuring distance on maps and globes, finding places by using latitude and longitude, etc.) [geography] ----------------------------------------------------------------------------
Instructional Practices. Student learning is not based solely on exposure to content; teachers facilitate learning through many instructional practices and experiences. A few instructional practice questions cut across subject areas. Typically, these are grouping students according to ability, amount of instructional time, and homework. Most instructional practice questions posed to teachers, however, are subject specific. To gather information about these subject-specific practices and experiences, teachers are asked questions such as the following:
---------------------------------------------------------------------------- Sample Instruction Practice Questions How often do you ask students to explain or support their understanding of what they have read? [reading] How often do you engage your class in using primary historical documents, letters, diaries or essays written by historical figures? [U.S. history] How often do you take your class on field trips? [geography] ----------------------------------------------------------------------------
Teacher Background, Education and Resources. NAEP has identified the following characteristics that may differ among teachers: type of certification, highest academic degree, undergraduate and graduate major and minor, undergraduate and graduate courses, opportunities for staff development, and teaching experience. While credentials and staff development
---------------------------------------------------------------------------- Sample Teacher Background, Education and Resource Questions Do you have teaching certification in any of the following areas that is recognized by the state in which you teach? What were your graduate major fields of study? During the past five years, have you participated in courses or workshops in any of the following? ----------------------------------------------------------------------------
Table 1's second dimension presents the purposes of the information gathered by the teacher background questionnaire. The NAEP teacher background questionnaire makes it possible to provide descriptive data about existing educational conditions, instructional factors by demographic subgroup, and trends in instruction. Also, information from the questionnaire provides opportunities to explore relationships between performance and instructional factors that may be associated with student achievement.
Existing instructional conditions. NAEP can inform educational policy by collecting descriptive background information about existing educational conditions, including instructional content, instructional practices, and teacher background, education, and resources. For example, it is possible to investigate existing teaching practices in geography by asking teachers questions such as "How often do students go on field trips that relate to the study of geography?
Instructional factors by demographic subgroups. NAEP collects information on basic demographic variables including sex, region, race-ethnicity, type of community, type of school and parental education. NAEP can tell us how aspects of education considered important are distributed among different groups of students. For example, "Are students in rural communities typically taught by more experienced teachers?"
Trends in instruction. A major purpose of the NAEP program is to measure change in student performance over time among groups of students. For example, "In what ways have reading materials changed over time?"
Comparative analyses of performance over two or more assessment cycles help engage the nation in discussions concerning the condition of American education. These discussions, in turn, provide a context for considering educational policy issues that may result in improvements in educational practices.
Relation of instruction to performance. Besides the descriptive data, the background variables collected by NAEP are useful in providing an overview of the most significant relationships between performance and various background factors across schools for the nation as a whole and among groups of students. For example a researcher may ask "Is there a relationship between assigning students to reading classes by ability and their overall reading performance?"
Changes in performance in relation to trends in instruction. Finally, the NAEP data may reveal differences in patterns of achievement, given changes in instructional content, instructional strategies, and teacher characteristics. For example, "Is the increase in the use of computers associated with higher average proficiency scores?"
Yet analyzing relationships between performance and the teacher's background, instructional content emphasis, and instructional practices does not reveal the underlying causes of the relationships. That is, without further investigation, it is impossible to know if, for example, certain instructional approaches have resulted in higher or lower performance. Nevertheless, the analysis of the NAEP assessment data using background variables can be informative when used in light of other knowledge from educational research.
Grade 4-Reading
1. How often do you do the following things as a part of reading instruction with this class?
a. Discuss new or difficult vocabulary.
b. Ask students to talk to each other about what they have read.
c. Ask students to work in a reading workbook or on a worksheet.
d. Ask students to write about what they have read.
e. Ask students to do a group activity or project about what they have read.
f. Ask students to read aloud.
g. Ask students to read silently.
h. Give students time to read books they have chosen themselves.
i. Give reading quizzes or tests.
j. Ask students to discuss different interpretations of what they have read.
k. Ask students to explain or support their understanding of what they have read.
Grade 4-U.S. History
1. About how many weeks during the school year do you spend covering the following?
a. Beginnings through the Revolution (e.g., colonization, settlement, revolution).
b. The new Nation through the Civil War (e.g., expansion, reform, Crisis of the Union).
c. The development of modem America (e.g., Reconstruction, industrial growth, U.S. role in world affairs, the Great Depression, the First and Second World Wars, immigration).
d. Contemporary America (e.g., civil rights movement, women's rights movement, Korean and Vietnam Wars, environmental movement).
2. About how much do you emphasize the following historical theories?
a. The development of American democracy and politics.
b. The gathering and interactions of peoples, ideas, and cultures.
c. The impact of economic and technological change.
d. The changing role of the United States in the world.
Grade 4-Geography
NAEP 1994 Reading Report Card, NCES 95-045. The complete, 179-page report. Includes all tables, explanatory data and an extensive collection of sample questions. Single copies available free from the National Center for Education Statistics, U.S. Department of Education, Washington, DC 20208-5653. Copies may also be obtained over the World Wide Web at http://nces.ed.gov/nationsreportcard/.
NAEP 1994 U.S. History Report Card, NCES 96-085. The complete, 155-page report. Includes all tables, explanatory data and an extensive collection of sample questions. Single copies available free from the National Center for Education Statistics, U.S. Department of Education, Washington, DC 20208-5653. Copies may also be obtained over the World Wide Web at http://nces.ed.gov/nationsreportcard/.
NAEP 1994 Geography Report Card, NCES 96-087. The complete, 122-page report. Includes all tables, explanatory data and an extensive collection of sample questions. Single copies available free from the National Center for Education Statistics, U.S. Department of Education, Washington, DC 20208-5653. Copies may also be obtained over the World Wide Web at http://nces.ed.gov/nationsreportcard/.
Focus on NAEP is a series that briefly summarizes information about the ongoing development and implementation of the National Assessment of Educational Progress. The series is a product of the National Center for Education Statistics (NCES), Pat Forgione, Commissioner, and Gary W. Phillips, Associate Commissioner for Education Assessment. This issue was written by Sheida White.
For questions about the content of this report, please contact Sheida White at Sheida.White@ed.gov