Skip Navigation
small NCES header image
Toward Better Teaching: Professional Development in 1993-94 / Appendix C

Appendix C. Technical Notes

The Schools and Staffing Survey

The data source for this study was the 1993-94 Schools and Staffing Survey (SASS:93-94)-a nationally representative survey that collected public- and private-sector data on the nation's elementary and secondary teachers and their schools and districts. SASS:93-94 is particularly useful for analyzing the professional development of elementary and secondary school teachers, because it is the latest and most comprehensive nationally oriented data set available with information on teachers' participation in professional development.28 This survey represented the first time that information was collected in a large national survey on the types of professional development activities in which teachers participated, the focus of these activities, the amount of time teachers were engaged in various activities, and the ways in which schools or districts supported teachers' participation in professional development. In addition, since SASS links school, principal, and district surveys with the teacher survey, it enables researchers to study how teachers' participation in professional development varies across different types of schools and districts and how it varies according to the individual characteristics of teachers and principals.

Sample Design

The 1993-94 survey was the third in a series of cross-sectional surveys, following ones in 1990-91 and 1987-88. It consisted of four sets of linked questionnaires, including surveys of schools, principals of selected schools, a subsample of teachers within each school, and public school districts. Stratified by state, sector, type, and association membership and grade level (for private schools); schools were sampled first. Each selected school received a school questionnaire and an administrator questionnaire. Within each school, a sample of teachers was selected and each one received a teacher questionnaire. Also, a Teacher Demand and Shortage questionnaire was sent to the local education agency (LEA) to collect information about school district's student enrollment, number of teachers, and hiring and retirement policies. A total of 13,271 schools and administrators, 68,284 teachers, and 5,459 LEAs participated in the 1993-94 survey.29

Data Collection Timing and Response Rates

Data collection for the 1993-94 SASS took place during the 1993-94 school year. The first mailing of questionnaires to teachers took place in January and February 1994 and the second in February and March. Telephone follow-up of mail nonrespondents took place between March and June.

The effective response rates (taking into account school response rates) were 84.7 percent for public school teachers and 72.9 percent for private school teachers. In the public school teacher survey, 91 percent of the items had a response rate of 90 percent or more; and in the private school teacher survey, 89 percent of the items had this level of response. None of the items used had a response rate of less than 75 percent. Values were imputed for questionnaire items that should have been answered but were not.

Study Sample

Since this study was designed to investigate teachers' professional development at the elementary and secondary levels, we excluded teachers who taught only prekindergarten or post-secondary classes. This resulted in a study sample of 55,118 elementary and secondary school teachers, including 46,916 public school teachers and 8,202 private school teachers. In order to take into account the different probability of selection of schools and teachers, as well as adjust for nonresponse and coverage bias, weights of the sample units (e.g., school weights or teacher weights) were developed in SASS to produce the estimates that were unbiased and consistent with estimates of national or state totals. Because the analysis unit in this study was teachers, we applied SASS teacher design weights for all of our analyses. Thus, the results of the study can be generalized to 1993-94 elementary and secondary school teachers in the United States.

Measures

The study emphasized five sets of outcome measures relevant to teachers' professional development and four sets of predictor measures describing characteristics of teachers, principals, schools, and districts (for public schools only). The specifics of how these measures were constructed, along with the SASS items from which they were drawn, are described below.

Outcome measures. Five sets of outcome measures describing teachers' professional development were investigated in the study. They included: 1) design of professional development (TT1020 and A835);30 2) delivery of professional development activities (T0545 to T0565, and T0700 and T0705); 3) content or topics of programs (T0590, T0600, T0610, T0620, T0630, and T1580) and duration of programs on various topics (T0595, T0605, T0615, T0625, and T0635); 4) professional development outcomes as perceived by teachers (T0640 to T0660); and 5) school context for teacher participation in professional development, including support provided to teachers (T0665 to T0690), and cooperative effort between teachers and principals (T1250, T1270, and T1290).

Predictor measures. These measures were used as classification variables in bivariate tabulations of the outcome measures described above or as predictor variables in multivariate analyses of the outcome measures. We focused on four sets of measures that described the following characteristics of teachers, principals, schools, and districts.

A. Teacher Characteristics

Teacher Level. Teachers were classified as elementary or secondary on the basis of the grades they taught (T0710 to T0785) rather than on the schools in which they taught. An elementary school teacher was one who, when asked for the grades taught, checked:

  • Only "ungraded" and was designated as an elementary teacher on the list of teachers provided by the school; or
  • Sixth grade or lower, or "ungraded," and no grade higher than sixth; or
  • Sixth grade or lower and seventh grade or higher, and reported a primary assignment of prekindergarten, kindergarten, or general elementary; or
  • Seventh and eighth grades only, and reported a primary assignment of prekindergarten, kindergarten, or general elementary; or
  • Sixth grade or lower and seventh grade or higher, and reported a primary assignment of special education and was designated as an elementary teacher on the list of teachers provided by the school; or
  • Seventh and eighth grades only, and reported a primary assignment of special education and was designated as an elementary teacher on the list of teachers provided by the school.

A secondary school teacher was one who, when asked for the grades taught, checked:

  • "Ungraded" and was designated as a secondary teacher on the list of teachers provided by the school; or
  • Sixth grade or lower and seventh grade or higher, and reported a primary assignment other than prekindergarten, kindergarten, or general elementary; or
  • Ninth grade or higher, or ninth grade or higher and "ungraded"; or
  • Seventh and eighth grades only, and reported a primary assignment other than prekindergarten, kindergarten, general elementary, or special education; or
  • Seventh and eighth grades only, and reported a primary assignment of special education and was designated as a secondary teacher on the list of teachers provided by the school; or
  • Sixth grade or lower and seventh grade or higher, or seventh and eighth grades only, and was not categorized above as either elementary or secondary.

Main Assignment field. Teachers' responses to items asking for their main and other assignment fields (T0315 and T0330, respectively) were aggregated into eight categories as follows:

  • Kindergarten/general elementary-Kindergarten or general elementary;
  • Mathematics or science-Computer science, mathematics, physical science, biology/life science, chemistry, geology/earth science/space science, physics, general or all other science;
  • English/language arts-English/language arts or reading;
  • Social studies-Social studies/social science (including history);
  • Special education-Special education, general, emotionally disturbed, mentally retarded, speech/language impaired, deaf and hard-of-hearing, visually handicapped, orthopedically impaired, mildly handicapped, severely handicapped, specific learning disabilities, or other special education;
  • Bilingual/ESL-Bilingual education or English as a second language;
  • Vocational education-Accounting, agriculture, business, marketing, health occupations, home economics, industrial arts, technical, or other vocational/technical education; and

  • Other-American Indian/Native American studies, art, basic skills and remedial education, dance, drama/theater, gifted, journalism, military science, music, philosophy, physical education, health, religion, French, German, Latin, Russian, Spanish, other foreign language, or all others.

Teaching status. Teaching status was classified into two categories-part time and full time-based on teachers' responses to items asking them to report the activity in which they spend most of their time (T0020) at the school or the amount of time they work as a teacher (T0030). Part-time teachers were those who reported working less than full time as a teacher at their school.

Teaching experience. This measure was a sum of total number of years that teachers taught full time and part time in public and private schools (T0095 to T0110). The sum was further classified into four categories: 0-3 years, 4-9 years, 10-19 years, and 20 or more years.

Highest degree earned. This measure was drawn from teachers' responses to items asking them about the type of education degree they had earned (T0170, T0235, T0270, T0285, and T0300). The measure was further classified into four categories: bachelor's degree or less, master's degree, educational specialist, and doctorate or first-professional degree.

B. Principal Characteristics

Highest degree earned. This measure, like the one above for teachers, was drawn from principals' responses to items asking the type of education degree they had earned (A060, A125, A160, A175, and A190). The measure was again classified into four categories: bachelor's degree or less, master's degree, educational specialist, and doctorate or first-professional degree.

Years of experience as a principal. This measure was a sum of the total number of years that principals reported serving as a principal in their current school (A325) and in other schools (A330). It was further classified into four categories: 0-3 years, 4-9 years, 10-19 years, and 20 or more years.

C. School Characteristics

Sector. This measure identified public schools and private schools. A public school was defined as an institution that provides educational services for at least one of grades 1-12 (or comparable ungraded classes), has one or more teachers who provide instruction, is located in one or more buildings, receives public funds as primary support, has an assigned administrator, and is operated by an education agency. Schools in juvenile detention centers and schools located on military bases and operated by the Department of Defense were included.

A private school was defined as a school not in the public system that provides instruction for any of grades 1-12 where the instruction was not provided exclusively in a private home. In order to be included in SASS, a school was required to provide instruction to students in at least one of grades 1-12 and not out of a private home. If it could not be determined whether or not it operated in a private home, the school had to have at least 10 students or more than one teacher. Schools that taught only prekindergarten, kindergarten, or adult education were not included.

Size. Size categories were based on the number of students (in head count) who were enrolled in grades K-12 in the school on or about October 1, 1990 (S0255). Size was recoded into four categories: less than 150 students, 150-499 students, 500-749 students, and 750 students or more.

Percent minority enrollment. This measure was the proportion of a school's total enrollment who were American Indian or Alaskan Native (S0405); Asian or Pacific Islander (S0410); Hispanic (S0415), regardless of race (Mexican, Puerto Rican, Cuban, Central or South American, or other culture or origin); and Black (0420) (not of Hispanic origin). Based on this proportion, the schools were further classified into five categories: 0 percent, 1-10 percent, 11-30 percent, 31-50 percent, and more than 50 percent.

Percent free/reduced-price lunch recipients. The proportion of students who received free or reduced-price lunch was computed for public schools that participated in the National School Lunch Program (S1680). Because relatively few private schools participate in the program, this variable was not computed for private schools. The proportion was recoded into four categories: 0-10 percent, 11-20 percent, 21-40 percent, and more than 40 percent.

Region. States were divided into four regions as follows:

  • Northeast-Maine, New Hampshire, Vermont, Massachusetts, Rhode Island, Connecticut, New York, New Jersey, and Pennsylvania;
  • Midwest-Ohio, Indiana, Illinois, Michigan, Wisconsin, Minnesota, Iowa, Missouri, North Dakota, South Dakota, Nebraska, and Kansas;
  • South-Delaware, Maryland, District of Columbia, Virginia, West Virginia, North Carolina, South Carolina, Georgia, Florida, Kentucky, Tennessee, Alabama, Mississippi, Arkansas, Louisiana, Oklahoma, and Texas; and
  • West-Montana, Idaho, Wyoming, Colorado, New Mexico, Arizona, Utah, Nevada, Washington, Oregon, California, Alaska, and Hawaii.

Community type. Community type was derived from the seven-category "urbanicity" code (locale) developed by Johnson.31 The locale code was based on the school's mailing address matched to Bureau of the Census data files containing population density data, Standard Metropolitan Statistical Area (SMSA) codes, and a Census code defining urban and rural areas. This code, also used in the 1990-91 and 1993-94 editions of Schools and Staffing in the United States: A Statistical Profile, is believed to provide a more accurate description of the community than the respondent's reported community type used in the 1987-88 edition of Schools and Staffing in the United States. For this study, the seven locale codes were aggregated into the following three community types:

  • Central city-A large central city (a central city of an SMSA with population greater than or equal to 400,000 or a population density greater than or equal to 6,000 per square mile), or a mid-size central city (a central city of an SMSA, but not designated as a large central city);
  • Urban fringe/large or large town-Urban fringe of a large or mid-size city (a place within an SMSA of a mid-size central city and defined as urban by the Bureau of the Census), or a large town (a place not within an SMSA, but with a population greater than or equal to 25,000 and defined as urban by the Bureau of the Census); and
  • Rural/small town-Rural area (a place with a population of less than 2,500 and defined as rural by the Bureau of the Census) or a small town (a place not within an SMSA, with a population of less than 25,000, but greater than or equal to 2,500, and defined as urban by the Bureau of the Census).

Private school affiliation. This measure was drawn directly from the SASS School survey (AFFIL). It has three categories: Catholic, other religious, and nonsectarian.

D. District Characteristics

A public school district (or LEA) was defined as a government agency administratively responsible for providing public elementary instruction, secondary instruction and educational support services, or both. The agency or administrative unit was required to operate under a public board of education. Districts that did not operate schools but that hired teachers were included. A district was considered out of scope if it did not employ elementary or secondary teachers of any kind, including special education and itinerant teachers.

District size. Public school district size categories were based on the number of students (by head count) who were enrolled in the district on or about October 1, 1987 (as reported in Item #1 on the Teacher Demand and Shortage Questionnaire, D0255). The count was recoded into four categories: less than 1,000 students, 1,000-4,999 students, 5,000-9,999 students, and 10,000 students or more.

Statistical Procedures

The major issues investigated in this study fell into five categories: 1) the influence of various groups in determining the content of in-service programs; 2) the participation rate of teachers in professional development activities; 3) the content and duration of these activities; 4) the school context; and 5) the outcomes of participation in professional development. To address these issues, bivariate analyses to examine the overall pattern of teachers' participation in professional development were conducted. Then multivariate analyses were conducted to explore how this participation varied by different kinds of teachers and teachers in different types of schools and districts.

Bivariate analysis. In this part of analysis, we examined overall patterns of teachers' participation in professional development, including their rates of participation in various types of professional development activities, their participation rate in programs that focused on various topics, the amount of time teachers were engaged in these activities, the school context for professional development, and how teachers assessed the effectiveness of the programs and their teaching practices. We also examined principals' perceptions of the influence of various groups on determining the content of in-service programs. Because of the differences between the professional development delivery mechanisms of public and private schools, we analyzed public and private teacher and school data on professional development separately.

In addition to examining the overall patterns of teacher participation in professional development, we also conducted a series of bivariate comparisons between different kinds of teachers (e.g., part-time versus full-time teachers) and teachers in different types of schools (e.g., those in small schools versus those in large schools). The comparisons were tested by the conventional Student's t statistic to ensure that the differences between the two groups of teachers were larger than that might be expected due to sampling variation. Tests for multiple comparisons were adjusted by the Bonferroni procedure, because when multiple statistical comparisons are made, it becomes increasingly likely that an indication of a population difference is erroneous. Generally, the Bonferroni procedure corrects the significance (or alpha) level for the total number of comparisons made within a particular classification variable. For each classification variable, there are (K*(K-l)/2) possible comparisons (or non-redundant pairwise combinations), where K is the number of categories. For example, highest degree earned by teachers has four categories (bachelor's degree or less, master's degree, educational specialist, and doctorate or first-professional degree). Thus, K = 4 and there are (4*3)/2 = 6 possible comparisons among the categories. The Bonferroni procedure divides the alpha level for a single t test (e.g., .05) by the number of possible pairwise comparisons in order to produce a new alpha that is corrected for the fact that multiple contrasts are being made.

Multivariate analysis. Although the bivariate analysis was important, it did not reveal the degree to which each teacher, principal, school, or district characteristic was related to teachers' participation in professional development, because many of these characteristics are often interrelated. To obtain a better understanding of teachers' participation in professional development, multivariate analyses were conducted to determine the unique importance of each teacher, principal, school, and district characteristic associated with this participation, net of other associations. WESVAR-PC was used to conduct the multivariate analyses. WESVAR-PC is a program that computes estimates and replicate variance estimates for data collected using complex sampling and estimation procedures.

Among all of the outcome variables examined in the bivariate analysis, we selected the following to examine with multivariate analytic techniques: 1) participation of teachers in various types of professional development activities; 2) participation of teachers in professional development programs focusing on various topics; 3) teachers' assessment of how the level of participation in professional development affected teaching practices. Controlling for various teacher, principal, school, and district characteristics, multivariate analysis allowed us to examine in detail who participated in professional development activities, who participated in programs that focused on various topics, and whether the level of participation had a significant impact on teaching practices according to teachers' perceptions.32

The first two questions, which have dichotomous outcomes (participated versus not participated in a particular professional development activity; and participated versus not participated in a professional development program on a particular topic), require logistic regression models. These models included the teacher, principal, school, and district characteristics described above as predicted variables. The full logistic regression in each case may be symbolized by the following mathematical equation:

Prob (Yes on Y)

log [ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ] = B0 + å BiXi + å BjXj + å BkXk + å BlXl + e

Prob (No on Y)

Where:

  • Y is teacher participation (yes or no) in one of the five professional development activities or in programs on one of the five program topics;
  • B0 is the intercept;
  • Bi is the effect of various teacher characteristics, Xi , on the log odds ratio of Y. Xi represents a series of teacher-level characteristics: main assignment field, teaching status, years of experience, or highest degree earned;
  • Bj is the effect of principal characteristics, Xj, on log odds ratio of Y. Xj represents two principal characteristics: principal's years of experience or highest degree earned;
  • Bk is the effect of school characteristics, Xk, on log odds ratio of Y. Xk represents a series of school characteristics: school size, percent minority enrollment, percent free/reduced-price lunch recipients, region, or community type; and
  • Bl is the effect of district characteristics, Xl, on log odds ratio of Y. Xl is district size.

The proportion of variance explained in each equation is indicated in appendix tables B1 and B2.

An ordinary least squares regression (OLS) was used to explore the effect of the level of participation on teachers' assessment of the effectiveness of professional development programs. Besides the level of participation, the model also included teacher, principal, school, and district characteristics as controls. We entered these variables in two steps: beginning with the level of participation, and then using the characteristics of teachers, principals, schools, and districts as statistical controls. This procedure was meant to quantify the unique impact of the level of
participation on the effectiveness of professional development programs in which teachers
participated, net of teacher, principal, school, and district characteristics. The full OLS regression model may be symbolized by the following mathematical equation:

Y = B0 + B1X1 + å BjXj + e

Where:

  • Y is teacher's assessment of the impact of professional development programs on teaching practices;
  • B0 is the intercept;
  • B1 is the effect of the level of participation, X1, on Y; and
  • Bj is the effect of a series of controls, Xj, on Y. Xj includes the characteristics of teachers, principals, schools, and districts listed above.

The proportion of variance explained in each equation is indicated in appendix tables B1 and B2.

Adjusted Values (tables 8a, b and 12a, b). The adjusted difference was computed by a mean-plugging procedure. The following examples illustrates the procedure. First, a logistic regression is run with a dummy outcome variable of whether teachers participated in workshops sponsored by school, Y, and four independent variables-teacher's level (X1), employment status (X2), teaching experience (X3), and educational attainment (X4). B0 is the intercept and B1 to B4 are the corresponding regression coefficients for independent variable X1, to X4. The logistic regression can be symbolized by the following mathematical equation:

Y

log [ ¾ ¾ ¾ ¾ ¾ ¾ ] = B0 + B1X1 + B2X2 + B3X3 + B4X4 + e

1 - Y

Next, the means (M1 to M4) are computed for these independent variables based on the teacher sample included in the study. To obtain the adjusted difference in participation rate between elementary and secondary school teachers (i.e. X1, and "0" coded for elementary school teachers and "1" coded for secondary school teachers), the means of the other three independent variables are plugged in the regression and the adjusted participation rate for elementary and secondary school teachers computed separately:

 

Y0 (for X1=0) e B0 + B2M2 + B3M3 + B4M4 / (1 + e B0 + B2M2 + B3M3 + B4M4);

Y1 (for X1=1) e B0 + B1 + B2M2 + B3M3 + B4M4 / (1 + e B0 + B1 + B2M2 + B3M3 + B4M4);

Finally, the adjusted difference in participation rate between elementary and secondary school teachers is computed by Y1 - Y0.

Estimate of Standard Errors

Since all estimates reported in this study were based on a sample rather than a population, they may differ somewhat from the figures that would have been obtained if a complete census had been taken using the same survey instruments, instructions, and data collection procedures. Generally speaking, there are two types of errors possible with an estimate based on a survey sample: nonsampling errors and sampling errors. Nonsampling errors can be attributed to many sources, such as the inability to obtain information about all cases in the sample, differences in how questions are interpreted, respondents' inability or unwillingness to provide correct information, or errors made in processing the data, and so on.

Sampling errors are attributed to sampling variation-that is, the variation that occurs by chance because a sample, rather than a population, is surveyed. It is primarily measured by a standard error that describes the reliability and accuracy of an estimate. It is essential to estimate the standard error for a statistic in a study based on a sample, because doing so enables researchers to construct confidence intervals, test hypotheses, and determine the precision obtained in a particular sample.

Because the SASS sample design involved stratification, clustering, unequal selection probabilities, and multistage sampling, the resulting statistics are more variable (i.e., have larger standard errors) than they would have been if they had been based on data from a simple random sample of the same size. Calculation of standard errors requires procedures that are markedly different from the ones used when the data are from a simple random sample. Popular statistical packages, such as SPSS or SASS, do not take complex sample design into account when they calculate standard errors. Along with a set of replicate weights supplied by SASS, we used the MPR-produced SAS procedure, REPTAB, which used a bootstrap variance estimator to estimate proportions and their standard errors for the bivariate analysis in this study.


FOOTNOTES:

[28] Several recent working papers published by NCES suggest improvements of questionnaire items and data collection. See U.S. Department of Education, , Kasprzyk, D., Measures of Inservice Professional Development: Suggested Items for the 1998-1999 Schools and Staffing Survey. Working Paper No. 96-25. U.S. Department of Education, , Rollefson, M., Student Learning, Teaching Quality, and Professional De-velopment: Theoretical Linkages, Current Measurement, and Recommendations for Future Data Collection. Working Paper No. 96-28.

[29] For a detailed description of the sample design, see Abramson et al. (1996)..

[30]Numbers in parentheses refer to the SASS Questionnaire items.

[31] F. Johnson, Assigning Type of Locale Codes to the 1987-88 CCD Public School Universe, Technical Report, Data Series: SP-CCD-87188-7.4, CS 89-194 (Washington, DC: U.S. Department of Education, , 1989); F. Johnson, "Comparisons of School Locale Setting: Self-Reported Versus Assigned" Working Paper No. 94-101 (Washington, DC: U.S. Department of Education, , 1994).

[32] The level of participation combined three elements: 1) teachers' participation in professional development programs on vari-ous topics; 2) the length of the program; and 3) the number of the programs in which teachers participated. This measure was constructed as follows: 1) we multiplied teachers' participation in programs on each of the five topics by the length of the pro-gram, and 2) we summed these products across the five topics. Bivariate results (see the attached table) suggested that the level of participation was positively associated with teachers' assessment of effectiveness of professional development programs. The higher the level of participation, the more likely were teachers to agree that these programs provided them with new informa-tion, changed their views on teaching, caused them to change their teaching practices, and made them seek further informa-tion or training; and the less likely were they to agree that these programs wasted their time.


Helpdesk Top of Page

Appendix B: Regression Results Contents List of Tables List of Figures

Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education