II. Methodology

The goal of this research was to determine whether teacher earnings data are currently collected by states and whether these data are comparable across states. To that end, the following steps were followed. First, a review was conducted of responses from a concurrent NCES project concerning teacher pay data collected by SEAs. Second, a search was conducted for relevant technical documentation on agency websites. Third, follow-up telephone discussions were conducted with relevant SEA contacts. Finally, the results and conclusions were submitted to the SEAs for validation. These steps are described in more detail below.

State education agencies annually report data to NCES on all public schools, public school districts, and the agencies themselves through the NCES Common Core of Data (CCD). SEAs exist for all 50 states, the District of Columbia, Puerto Rico, and the four outlying areas of American Samoa, Guam, the Commonwealth of the Northern Mariana Islands, and the U.S. Virgin Islands. This report limits the focus to those SEAs from the 50 states and the District of Columbia. Reporting by SEAs to NCES is voluntary.

NCES Common Core of Data Feasibility Study

NCES contracts with SEAs for their input to the quality and improvement of CCD data. Related to this, in July 2006 NCES sought feedback from SEAs on the availability of teacher level data required to compute comparable average teacher salaries within and across states. To this end, NCES asked 14 questions about the availability of 17 data elements for each certified teacher (see appendix A). The data elements included teacher characteristics as well as compensation information (e.g., base annual salary, retirement contributions from the district/state). The results from the feasibility study are used as a starting point for this report.

Internet Search for SEA Technical Documentation

Internet searches were used to confirm or elaborate upon results obtained from the CCD feasibility study and the follow-up discussions, as well as to identify any additional sources of data. All SEAs have websites with general organizational information and access to publications. The online search took place immediately prior to contacting each SEA by telephone and was used to identify materials that describe and specify details of district data reporting requirements and procedures. These materials often include data dictionaries and downloadable forms that provide information on the types of data that school districts report to state agencies. Although many SEAs do not make electronic documentation on data collection publicly accessible,⁵ the online search provided a useful complement to the other methods of gathering information. The information provided through the online search was useful for determining the consistency of SEA definitions for data items. In addition, in the case of two states (Maryland and North Dakota), the researcher was directed to specific documentation on the SEA website that formed the basis of the data collected for this report.

Top

Telephone Follow-up Discussions With NCES Data Contacts

While the results from the CCD feasibility study provided a great deal of information, it was not designed specifically to answer the questions that are the focus of this study. Also, a number of SEAs did not respond to the study. Thus, an additional data collection was necessary to obtain data from the nonresponding SEAs, as well as to supplement information from the CCD feasibility study with the additional items that are the focus of this research. For SEAs that provided information through the CCD feasibility study, the contact persons were recontacted to schedule follow-up discussions. SEAs that did not participate in the feasibility study were contacted through the CCD data coordinator information posted on the NCES website. (See appendix A for an example of the e-mail sent to the two groups of SEAs.) Telephone discussions occurred between October 2006 and March 2007.

A protocol was developed to structure the telephone discussions (see appendix A). Each conversation verified whether the state agency collects employment and salary data from school districts for individual teachers.⁶ SEAs that do not collect individual teacher data were asked to indicate when, if ever, they plan to collect such data and to describe the calculation process used to generate average teacher salaries reported for their state.⁷ Other questions sought more detailed information about critical variables necessary for calculating comparable measures of teacher compensation, including job titles, full-time equivalency (FTE), the nature of salary variables (total, base, and supplemental pay), teaching experience, and educational attainment. Some SEAs provided additional information from other sources that was included with data collected directly through the telephone discussion.⁸

Validation of Results by SEAs

A preliminary draft of this report was sent to all data contacts in late March, 2007. Contacts were asked to validate the information reported for their state, with a particular focus on the information presented in table C-2 (see appendix A for an example of the letter accompanying the draft of the report that was sent to SEAs). Twenty-three SEAs provided feedback (see table B-1), with a small number of recommended corrections to table C-2.

Top

Summary

Collecting data about teacher pay from multiple sources allowed researchers to confirm results, clarify questions about responses to the CCD feasibility study, and, in general, substantiate whether each SEA collects teacher compensation data and the characteristics of specific variables. The telephone discussions and the documents found on the Internet were especially helpful for probing more deeply about variables that are essential to comparing teacher pay and compensation across states; the conversations were the only means available for inquiring how researchers might gain access to SEA-maintained data.

Response Rates

SEAs varied in their participation with the different collection methods. Response rates ranged from 61 percent for the CCD feasibility study to 45 percent for the SEA validation request. However, by taking advantage of multiple approaches, reasonable coverage was achieved. Overall, 82 percent of the 51 SEAs included in this research responded to at least one of the data collections (including having information available on their website). Forty of the 51 SEAs (78 percent) responded to either the CCD feasibility study or the telephone discussion, and 22 (43 percent) responded to both. One state did not participate in the CCD feasibility study, the telephone discussion or the SEA validation, but information was found on the SEAs website. One state did not participate in the CCD feasibility study, the telephone discussion, and no information was found on the SEA website; however, this SEA did respond to the validation request. Only 9 SEAs (18 percent) did not respond to the data collections.

Reconciling Data Across Collection Methods

As one would expect when data are collected on the same topic using different collection methods, results within a state were occasionally inconsistent. Inconsistencies could occur because different individuals interpreted the questions differently, slightly different questions were asked, facts changed between collection efforts, or for other reasons. Responses from the CCD feasibility study and the telephone discussion were generally consistent. In situations where the information from the telephone discussion could not be fully clarified (i.e., "undetermined" status in tables C-2 and C-3), information from the feasibility study took precedence. The SEA validation phase also provided another opportunity to resolve data inconsistencies. SEAs generally provided explanations for their recommended changes, and the changes were incorporated into this report.⁹

The SEA' provided some inconsistent responses to questions on data availability on the CCD feasibility study and telephone discussion. There are several possible reasons why these different responses were given in the two data collections. First, the wording of the question on data availability in the CCD feasibility study and the telephone discussion is different. In the feasibility study SEA contacts were asked, "When would you be able to send us a complete data file with all of the data items requested?"¹⁰ In the discussion, respondents were asked, "Would you be able to release a research version of the teacher data that does not permit identification of individual teachers, but that can be used to analyze teacher turnover, mobility, pay and other labor market factors?"¹¹ Other explanations for these inconsistencies could include that the information was collected at different points in time or that the respondents to the two collections were not the same person, and one or neither of the respondents may be the individual in charge of determining data availability. Despite these inconsistencies, the individual and overall determinations of availability do provide meaningful information about the likelihood of the SEA to make these data available to NCES and/or the research community.

Top

---

⁵ Relevant online data documentation was located for 24 states.
⁶ See exhibit A-1 for the definition of a teacher in the CCD feasibility study. The telephone follow-up discussion used a broader definition, allowing representatives to define the term themselves. Notable differences include that the CCD feasibility study specifically excluded teaching assistants, librarians, etc., and teachers without teaching certificates. Given the exploratory nature of the analysis, and the additional questions asked about data file details (including job title), it is not believed that these differences substantially affected the findings.
⁷ Some states that do not directly collect teacher-level salary data are able to report teacher salary data. For example, the California SEA collects teacher-level data other than salary and uses these data to impute estimated salaries from known salary rate tables.
⁸ As previously stated, two states directed the researchers to data documentation on the SEA websites; a third SEA sent data documentation.
⁹ Corrections to information on data availability, length of contract, and teacher identifiers and demographics were recommended by SEAs during the validation phase. Overall, there were relatively few discrepancies in the data or recommendations for changes made by the SEAs.
¹⁰ See exhibit A-1 for the listing of requested items in questions 2 and 4 of the CCD feasibility study.
¹¹ See exhibit A-4 for the protocol used during the telephone discussion.