Survey of Earned Doctorates (SED)

5. Data Quality and Comparability

Although the SED has evolved over its history, the types of information gathered on the survey questionnaire (i.e., demographic information, educational history, and post-graduation plans) have been relatively stable over time. The survey did not change substantially in the 15 years following its first fielding in 1957. Between 1973 and 2000, new questions were added to the survey that addressed disability status, number of years as a full-time graduate student, and debt levels at time of doctorate receipt. In the 1990s, the focus was on the evaluation of existing items and the use of advisory panels for recommendations on new items for the SED 2001, after which the SED experienced several variable, code frame, and format changes. Several new items were added to the instrument in 2001 including secondary dissertation field, amount of tuition remission, graduate entry, and time spent taking classes and working on dissertation. Additional items were modified to differentiate undergraduate debt from graduate debt and to request the number of dependents within respective age categories.

Beginning with the SED 2004, some federal sponsor-approved changes were made to the standard questionnaire: questions were added to gather data on additional post-secondary degrees, master’s degree as a prerequisite, and postdoc position. In addition, educational history items were redesigned and reformatted to ask only for information on completed degrees. Response codes for various items were also modified. The questionnaire was changed slightly again in academic year 2007, with the most substantive changes being an expansion of the code frame for undergraduate and graduate debt, the inclusion of both month and year for additional postsecondary degrees and for graduate school entry, the addition of an item identifying any time spent out of the doctoral program, the addition of a response option for postgraduate plans, and a modification  to collect only the last four digits of an individual’s social security number.

The SED 2008 questionnaire added items related to the respondent’s annual salary for the principal job reported for post-graduation plans. The other notable change in the 2008 questionnaire was reversing the question order for the educational grid; the “month and year granted” for each degree was asked before the “month and year started” for each degree on the educational grid, thus reversing the order from previous questionnaires.

The SED 2010 questionnaire added two items pertaining to the American Recovery and Reinvestment Act (ARRA) of 2009 as a source of financial support. The SED 2010 questionnaire also revised the disability item and the Hispanic ethnicity item by eliminating the screener questions (e.g., “Are you a person with a disability”) and rewording the question or response categories. For 2011, the SED was identical to the 2010 survey. After a 3-year review of the EdD degree programs participating in the SED, 77 programs were reclassified from research doctorate to professional doctorate in 2010 and another 66 programs were reclassified in 2011. Beginning with 2010, SED data are no longer being collected from graduates earning degrees from the reclassified EdD programs. The exact number of individuals who graduated with doctorates in the reclassified EdD programs is unknown. However, in 2009, 1,136 doctorate recipi­ents earned degrees from EdD degree programs that were reclassified in 2010. Of these doctorate recipients, 96 percent iden­tified their field of study as education, 2 percent reported a science and engineering field of study, and 2 percent identified a non-science and engineering field of study other than education.

Sampling and Nonsampling Error

Two general types of error affect surveys: sampling error and nonsampling error. Because the SED is a census of the population of research doctorate recipients, no error results from sampling; however, minimizing nonsampling error is a constant goal of the SED data collection. Nonsampling error affects the SED at the unit (survey) and item (question) levels.

Sources of unit-level error include: 1) coverage error, resulting from a failure to identify all eligible members of the target population, and 2) questionnaire or unit nonresponse resulting from a participant’s failure to or refusal to complete a questionnaire or a participant’s not having the opportunity to complete a questionnaire.

Sources of item or question-level error include: 1) item nonresponse (failure to provide data for particular items on a returned questionnaire); 2) measurement errors that occur when data collection methods or inconsistent interpretations of questions introduce bias; and 3) processing errors that occur during data entry, editing, or coding.

Coverage Error. The SED is administered to all research doctorate recipients identified by the universe of research doctorate-granting institutions. True coverage errors for this population result from one of two sources: (1) inaccurate specification of the institution universe (i.e., omitting institutions that grant research doctorates or doctorate-granting programs within institutions), or (2) a failure to fully enumerate the frame of research doctorate recipients.

Top

Given the high visibility and participation of doctorate-granting institutions, there is little, if any, coverage error resulting from the first source. Because the graduate schools keep accurate records of degree recipients and rarely refuse to participate, the second source, or frame of recipients, is also quite accurate. In 2009, for example, only two doctoral granting universities refused to participate, and the number of graduates at these two schools was very small for the academic year of 2009. Comparisons of the number of research doctorates covered by the SED with the total number of doctorates (including non-research doctorates) reported by institutions in the IPEDS system also confirm that coverage of research doctorates in the SED is excellent.

Survey contract staff compare the numbers of doctorates granted by each institution in the IPEDS database with the totals in the SED database. The numbers are not exactly comparable because IPEDS includes some non-research doctorates that are excluded from SED, but the comparisons do provide an alert to any major discrepancies in the basic counts and thus possible problems with the SED universe. Aside from the minimal amount of coverage error that may exist, the primary source of misclassification in the universe stems from institutions returning completed SED forms from individuals who will not, as it turns out, receive their degrees until the following academic year or individuals completing the web survey and then postponing their graduation to another year. Cross-round de-duplication procedures are performed to address this source of error.

Additional efforts to reduce error and to enhance the data collection process include requesting that the Institution Contact (IC) provide a final graduation list or an annotated commencement program used to document degree conferral. Many ICs cross off names of individuals appearing in the commencement program who failed to complete requirements necessary for degree award and add names of individuals who were not included in the program (due to the doctorate recipient’s request or administrative circumstance). When materials are received from institutions that do not annotate commencement programs, the survey contractor calls to verify the accuracy and completeness of the information. This comprehensive approach ensures that the number of missing doctorate recipients is likely to be quite small. An indication of the minutia of missing doctorate recipients is found in the number of cases that are added to the universe from late arriving questionnaires. In the 2009 data collection for example, 97 cases were discovered to be doctorate recipients from prior years and added to the universe of their respective rounds; these cases represent less than one-tenth of one percent of the universe for their respective years of data collection.

Nonresponse Error. Questionnaire or unit non-response occurs when doctorate recipients who are identified for participation in the survey fail to complete surveys. SED unit response rates represent the rates at which doctorate recipients do complete and return questionnaires. Table SED-1 displays overall respondent completion rates from the 1958 academic year through 2011.

The goal of the SED is to produce a stable respondent completion rate of 92 percent. For an institution to realize a response rate of 90 percent or better, experience shows that it must distribute and collect the instrument or the web URL to all eligible individuals and strongly encourage completion. For this reason, the survey contractor communicates an expectation to all institutions for at least a 90 percent respondent completion rate and works to help them find ways to realize and exceed that level. In 2009 for example, just under two-thirds of participating institutions had response rates of 90 percent or better among the 429 eligible research doctorate-granting institutions, with 155 schools that were below 90 percent.  The number of institutions that had response rates of less than 90 percent in academic year 2009 was slightly higher than in 2008.

Top

Table SED-1. Overall survey response rates, 1958-2011

Year	Self- report rate	Year	Self-report Rate
1958	93.3	1985	94.8
1959	97.9	1986	93.5
1960	97.8	1987	93.1
1961	97.3	1988	92.9
1962	97.6	1989	92.3
1963	97.4	1990	93.6
1964	96.9	1991	94.6
1965	97.4	1992	95.1
1966	97.6	1993	94.7
1967	96.3	1994	94.6
1968	97.3	1995	94.2
1969	97.6	1996	93.0
1958	96.6	1997	91.6
1970	93.6	1998	91.9
1971	92.3	1999	91.9
1972	90.2	2000	92.4
1973	88.5	2001	92.7
1974	83.9	2002	91.3
1975	90.7	2003	91.6
1976	91.2	2004	91.3
1977	91.4	2005	92.1
1978	91.0	2006	93.1
1979	91.0	2007	91.7
1980	96.2	2008	92.3
1981	95.7	2009	92.6
1982	95.3	2010	93.0
1983	95.5	2011	92.8
1984	95.1
NOTE: Rates for 1976-2010 include late responses. Rate for 2011 may increase slightly in the next year if additional questionnaires are received after survey closure.
SOURCE: National Science Foundation, National Center for Science and Engineering Statistics. (2010). Survey of Earned Doctorates Abbreviated Methodology Report 2009.  Unpublished report. Arlington, VA: Author.; National Science Foundation. (2012). Doctorate Recipients from U.S. Universities: 2011 (NSF 13-301). Arlington, VA: Author.

Unit nonresponse. Of the 49,010 individuals who obtained a research doctorates in 2011,  93 percent completed the SED. Records for nonrespondents are constructed from limited information (doctoral institution, year of doctorate, field of doctorate, type of doctorate, and, if available, baccalaureate institution, master's degree institution, and sex) collected from commencement programs, graduation lists, and other similar public records. These constructed records are not included in the self-report response rates listed in Table SED-1. Student nonresponse was concentrated in certain institutions. The 41 institutions with the highest percentage of students not responding in 2011 accounted for 63 percent of the total number of nonrespondents.

Item nonresponse. Item nonresponse rates in 2011 for the key SED demographic variables ranged from 0.04 percent for sex to 7.2 percent for location after graduation. No imputation was performed for missing data items.

Measurement Error.  Measurement error in the SED is attributable to several sources, including error in recording respondent data (calculated at less than 1 percent) and coding error for some variables due to the difficulty of defining some concepts (calculated at 0.34 percent). For example, an SED respondent may classify his or her field of specialty differently than the department or university does in its institutional reporting for the IPEDS Completions Survey.

Data Comparability

Because a prime use of the SED data is trend analysis, tremendous efforts have been made to maintain continuity of survey content. While both unit and item response rates in the SED have been relatively stable through the years, changes to the survey instrument can affect data comparability.

This may be especially important to consider when analyzing data by citizenship and race/ethnicity, where very small fluctuations in response options may result in increases or decreases in counts that do not reflect real trends. New procedures implemented in the early 1990s had a significant positive impact on response to these two items, as well as to the items on foreign country of citizenship and postdoctoral location, work activity, and employment field. Another potential comparability issue for the SED involves changes over time to the survey’s Specialties List, which is used to code fields for degrees, postdoctoral study, and employment. Readers are also cautioned that the 2010 data on education doctorates are not strictly comparable with data of previous years due to the reclassification of the EdD.

Top

The racial/ethnic question has undergone several revisions over the years. In 1977, it was modified to correspond to a standard question format recommended by the Federal Interagency Committee on Education and adopted by the Office of Management and Budget (OMB) for use in federally sponsored surveys. In 1980, the question was further revised; the Hispanic category was subdivided into Puerto Rican, Mexican American, and other Hispanic; and respondents were asked to check only one racial category. Prior to 1980, doctorate recipients could check more than one category to indicate their race. The item was modified again in 1982 to separate the questions on race and ethnicity. Currently, respondents are asked first to indicate whether they are Hispanic or Latino and then to select one or more of the five racial categories (American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, Asian, Black or African American, and White).

Major changes to questions pertaining to sources of support and debt (tuition and fees, living expenses and supplies, and transportation to and from school) were also implemented from the late 1990s to 2001.  Prior to 1988 there were 35 possible numeric values for SED source of funding variables.  Several of these codes were tied to specific Federal programs (e.g., Patricia Roberts Harris scholarships, NIH traineeships, etc.). The newer code frame reduced the respondent’s available choices to 13 and presented options as broad categories of funding sources rather than specific programs.
Users should take these changes into account when analyzing trends and consult the most current NSF procedures for the SED.

Comparisons with SDR.  Each year’s doctorate recipients provide information on postgraduation employment or study plans on the survey form. Since the questionnaire is filled out around the time the doctorate is awarded, a recipient’s plans are subject to change. However, comparisons with the longitudinal Survey of Doctorate Recipients (SDR) have shown SED data to be a reasonable indicator of actual employment status in the year following the doctorate, although results vary by sector.

Comparisons with IPEDS. The IPEDS Completions Survey also collects data on doctoral degrees, but the information is provided by institutions rather than by doctorate recipients. The number of doctoral degrees collected by the IPEDS Completions Survey is slightly higher than the number collected by the SED. This is primarily because the IPEDS Completions Survey collects data on both nonresearch and research doctorates, whereas the SED is limited to research doctorates. Differences in counts have been generally consistent since 1960, with ratios of IPEDS-to-SED counts ranging from 1.01 to 1.06. Because a respondent to the SED may not classify his or her specialty in the exact same way that the institution reports the field in the IPEDS Completions Survey, differences between the two surveys in the number of doctorates for a given field may be greater than the difference for all fields combined.