Appendix A: Methodology and Technical Notes

EDFacts Collection System

EDFacts is a U.S. Department of Education initiative to centralize and coordinate the administrative data reported by state education agencies (SEAs) to the Department of Education for elementary and secondary public education. Program offices within the Department sponsor specific portions of the data reported in EDFacts to meet information requirements and support program monitoring and policy development. The purpose of EDFacts is to

• place the use of robust, timely performance data at the core of decision and policymaking in education;


• reduce state and district data burden and streamline data practices;


• improve state data capabilities by providing resources and technical assistance; and


• provide data for planning, policy, and management at the federal, state, and local levels.

EDFacts provides the collection and processing systems that allow SEAs to report data annually for multiple elementary/secondary programs, such as the Common Core of Data and the Consolidated State Performance Report, through a series of data files that fall into different reporting schedules throughout each year. SEAs reported all the data elements used in this report for the adjusted cohort graduation rate (ACGR), averaged freshman graduation rate (AFGR), and the event dropout rate through the EDFacts Submission System.

For more information on the EDFacts initiative, please visit the public website at http://www2.ed.gov/about/inits/ed/edfacts/index.html.

Consolidated State Performance Report (CSPR)

The CSPR collection is stewarded and monitored by the Department’s Office of Elementary and Secondary Education (OESE). The CSPR is the required annual reporting tool for each state, the District of Columbia, Puerto Rico, and the Bureau of Indian Education (BIE) as authorized under Section 9303 of the Elementary and Secondary Education Act (ESEA), as amended.

Part I of the CSPR collects information required for the Annual State Report to the Secretary of Education, as described in section 1111(h)(4) of ESEA and data required under Homeless Collection (added in fiscal year 2005–06). Examples of data in Part I include: participation and performance on state assessments, participation and performance of English learners in language programs, highly qualified teachers, and homeless students served.

Part II of the CSPR collects information related to state activities and outcomes of specific ESEA programs needed for the programs’ Government Performance and Results Act indicators or other assessment and reporting requirements. OESE uses these data in conjunction with data collected in Part I to monitor states’ progress in implementing ESEA and to identify technical assistance needs and program management and policy needs. Examples of data in Part II include: participation in Title I Part A, migrant students served, neglected or delinquent students served, adjusted cohort graduation rates, and lists of identified schools.

The CSPR is considered OESE’s official report on state-level data for the specific programs included for a given school year. Figures published in this report may differ from those in related CSPRs for a given state. State CSPR reports include data submitted as of the final CSPR deadline. SEAs may update data beyond the CSPR deadline, and data in this report may reflect those updates. For more information about the CSPR, please e-mail questions to CSPR@ed.gov.

The Common Core of Data (CCD) Program

The CCD is a program of the National Center for Education Statistics’ (NCES) Administrative Data Division, which is part of the U.S. Department of Education’s Institute of Education Sciences. CCD was established as part of the Cooperative Education Statistics System in section 157 of the Education Sciences Reform Act of 2002, part C. Each school year the CCD program collects fiscal and nonfiscal administrative data about all public schools, public local education agencies (LEAs), and SEAs in the United States. The State Nonfiscal Survey of Public Elementary/Secondary Education includes the data elements used to calculate the AFGRs and the event dropout rates in this report and is one of six annual surveys that comprise the CCD. The other five surveys are the Public Elementary/Secondary School Universe Survey, the Local Education Agency Universe Survey, the National Public Education Finance Survey, the School District Finance Survey, and the Teacher Compensation Survey.

The objectives of the CCD are twofold: first, to provide an official listing of public elementary and secondary schools and LEAs in the nation, which can be used to select samples for other NCES surveys. And second, to provide basic information and descriptive statistics on public elementary and secondary schools and schooling in general that are comparable among states.

SEAs report CCD nonfiscal survey elements as part of the annual EDFacts collection. SEAs report CCD fiscal data through separate surveys that are conducted in collaboration with the U.S. Census Bureau. CCD contains three categories of information: general descriptive information on schools and school districts, data on students and staff, and fiscal data on revenue and expenditures for public education. CCD publishes statistical information annually by school year for approximately 100,000 public elementary and secondary schools, approximately 18,000 local education agencies (including independent charter districts, supervisory unions, and regional education service agencies) in the 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Department of Defense Education Activity (DoDEA), BIE, Guam, the Commonwealth of the Northern Mariana Islands, and American Samoa.

Data Collection and Review

This report and the accompanying data files provide both ACGRs and AFGRs. For the ACGR data, SEAs track a cohort of high school students over 4 years and then submit to EDFacts aggregated counts of students remaining in that cohort at the end of 4 years, the counts of students from that cohort who received a diploma at the end of the fourth school year, and the calculated ACGRs from these counts. For the components used to calculate AFGRs, SEAs submit to EDFacts each school year the October 1 membership counts and aggregate counts of the total number of students who graduated at the end of that school year. Both OESE and NCES run a series of validation checks against the reported data. Both NCES and OESE check reported data for internal, cross-level, and cross-year consistency. State coordinators are asked to review any identified anomalies and either update the reported data or explain the anomaly within a stipulated time period. In most cases states were able to correct or explain their data. SEAs are asked to respond to these edits and provide either data revisions or explanations for the identified anomalies.

Because data collected in the CSPR are used for program administration, OESE does not alter the data reported by SEAs as part of the CSPR. Data errors and anomalies must be resolved by the state. In some instances, OESE has not published state-submitted data due to data quality concerns. In cases where data are missing or suppressed, NCES worked with OESE to impute these data to improve the accuracy of national estimates and to make such estimates nationally representative.

NCES uses statistical editing procedures to identify inconsistencies or anomalies in reported values for CCD that are used to compute the AFGR. Critical data errors include cases where the counts at a lower-level of aggregation exceed counts at higher levels of aggregation, cases where a graduate count exceeds the enrolled population count by more than 5 percent, or cases where the current year data vary widely from data reported in prior years. One method for identifying inconsistencies involves looking at the consistency of the data for an individual institution over a 5-year period and comparing the mean variation across prior estimates to the variation between those prior data and data collected in the current year. NCES provides the results of these edit checks to SEAs and requests that SEAs verify and explain the flagged inconsistencies. Or if the SEA finds that the submitted data have errors, NCES asks that the SEA resubmit corrected data. If the data for an individual institution (school, LEA, or state) are flagged as failing several “critical” data checks and the state is unable to provide a detailed explanation of the anomaly, then the data, as reported to EDFacts, are not reported to the public. NCES does, on occasion, alter the data reported by SEAs for CCD. NCES adjusts, suppresses, and/or imputes the data in question to correct for internal and cross-level consistency. Unexplained violations of this check at the school or LEA level result in the suppression of the identified data point. Violations at the state-level can result in the suppression and imputation replacement of the identified data point. Specific information about the imputation methods applied can be found later in this appendix under the heading “Imputation Methodology.”

In some instances the reported SEA totals are less than the aggregates from the LEA or school level. In such cases where the SEA could not explain or correct this inconsistency, NCES applied a “raking” procedure to the LEA and/or school-level data to ensure that the sum of the school or LEA data is consistent with the state-level data. This “raking” process identifies the percentage of students at the school and/or LEA level that exceeds the state level and removes that percentage from each school and/or LEA. The raking algorithm uses a statistical rounding technique to carry forward any resulting remainders throughout the raking process in order to maintain whole number counts of students. This process results in slight changes to individual school/LEA records in an attempt to make the aggregate values more statistically consistent.

The EDFacts collection system accepted blank responses in SY 2010–11 and SY 2011–12 and did not require that states distinguish among missing, not applicable, and “zero” values. NCES used statistical editing procedures to change blank responses to missing, not applicable, or zero using information available from SEAs or from prior year reporting. However, it is possible that some blank responses may have been categorized incorrectly. For example, the number of graduates for a specific race group may have been categorized as missing when the actual count was zero.

Response and Nonresponse

ACGR. For both SY 2010–11 and SY 2011–12, forty-seven states, the District of Columbia, and BIE reported 4-year ACGRs. Idaho, Kentucky, Oklahoma, and Puerto Rico have approved timeline extension requests to delay reporting of the ACGR. Kentucky and Oklahoma will first report the ACGR on SY 2012–13. Idaho will first report the ACGR on SY 2013–14. These systems were granted extensions in order to provide time for their internal data systems to mature to the point when they would have the requisite data to calculate the ACGR components and rate. Puerto Rico reported year rate for SY 2011–12 and will continue to report a 3-year rate since Puerto Rico’s high school structure does not allow for reporting a 4-year rate. The estimated national ACGR includes reported data from 47 states and the District of Columbia and imputed data for Kentucky, Oklahoma, and Idaho. Detailed information on these imputations is provided later in this document in the section marked “Imputation Methodology.” Although DoDEA, the Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands are included in the collection for the CCD, these jurisdictions were not included in the collection of ACGR data for SY 2010–11 and SY 2011–12. Texas did not report the ACGR components or rate for SY 2011–12 by the CSPR reporting deadline but did successfully submit their data in time for inclusion in this report.

CCD Graduate Data. SEAs from 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands reported graduate counts in EDFacts for SY 2010–11. BIE, DoDEA, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands did not report any graduate counts to EDFacts for SY 2010–11. New Jersey did not report graduate counts disaggregated by gender for SY 2010–11. In order to produce a national estimated AFGR data by gender for SY 2010–11, NCES imputed the graduate counts disaggregated by gender for New Jersey.

SEAs from 49 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands reported in EDFacts for SY 2011–12. Texas, DoDEA, BIE, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands did not report any graduation data for SY 2011–12. New Jersey did not report graduate counts disaggregated by gender for SY 2010–11. In order to produce a national estimated AFGR data for SY 2011–12, NCES imputed the graduate counts for Texas and the graduate counts disaggregated by gender for New Jersey.

CCD Dropout Data. SEAs from 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands reported dropout counts in EDFacts for SY 2010–11. BIE, DoDEA, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands did not report dropout data to EDFacts for SY 2010–11.

SEAs from 48 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands reported dropout counts in EDFacts for SY 2011–12. Utah, Alabama, BIE, DoDEA, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands did not report any dropout data to EDFacts for SY 2011–12. In order to produce a national estimated event dropout rate for SY 2011–12, NCES imputed dropout data for Utah and Alabama.

NCES makes every effort to work with SEAs to allow them opportunities to submit, correct, and/or explain data before imputation procedures are used in place of reported data. NCES only imputed missing data that were necessary to produce United States totals from the 50 states and the District of Columbia. Data for the other nonrespondent jurisdictions that are not included in the United States level estimates were not imputed and are shown as missing in the report tables. Detailed information on the imputation methodology is provided below.

Imputation Methodology

Several imputations procedures, each based on the specific circumstances of the missing/suppressed data item, are employed.

Method 1: Carrying forward prior year rates

If a school system was unable to report data for the current year but had been able to report the data for a proceeding year, the rates for the previous year were applied to the current year data to estimate the missing item. The rates were carried forward by the lowest level of disaggregation and totals were derived from the imputed disaggregates.

• Missing Diploma Counts (numerator for the AFGR): Ratio of prior year diplomas to 12th- grade membership applied to current year grade 12 membership.


• Missing Cohort Graduates (numerator for the ACGR): Ratio of prior year cohort graduates to total graduates applied to current year total graduates.


• Missing Cohort Student Count (denominator for the ACGR): Ratio of prior year cohort student count to averaged freshman count (denominator for the AFGR) applied to current year average freshman count.

Method 2: Apply average state ratio to missing SEA

If an SEA did not report a component to the ACGR but did report the corresponding component used to calculate the AFGR, NCES computed a ratio of the ACGR to AFGR components from the SEAs that did report both these components. Then, for each state with a missing ACGR component, NCES applied that ratio to the AFGR component to derive the imputed value for the missing ACGR component. The ratios were derived at the lowest level of disaggregation, and totals were derived from the imputed disaggregates.

• Missing Cohort Graduates (numerator for the ACGR): Average (weighted) ratio of cohort graduates to total graduates across reporting states applied to total graduates in the target state.


• Missing Cohort Student Count (denominator for the ACGR): Average (weighted) ratio of the cohort student count to the averaged freshman count (denominator for the AFGR) across reporting states applied to averaged freshman count in the target state.


• Missing Cohort Rate Components by Disability, English Language Learner, and Poverty Status: Average (weighted) ratio of the overall ACGR components to the disaggregate subgroups across reporting states applied to either reported or imputed overall ACGR for target state.

Method 3: Carrying back current year rates

If a school system was unable to report data for the prior year but was able to report the data for the current year, the rates for the current year were applied to the prior year data to estimate the missing item. The rates were carried back by the lowest level of disaggregation and totals were derived from the imputed disaggregates.

• Missing Cohort Graduates (numerator for the ACGR): Ratio of current year cohort graduates to total graduates applied to prior year total graduates.


• Missing Cohort Student Count (denominator for the ACGR): Ratio of current year cohort student count to averaged freshman count (denominator for the AFGR) applied to prior year averaged freshman count.
  

Variability of Data Quality

SEAs and LEAs vary widely in how they collect and manage student data and in the degree of rigor applied to verifying the accuracy of graduation and dropout data at all levels. Because of this, the graduation and dropout data reported by SEAs may have varying levels of quality. Those states that collect dropout or graduation data through student-level records systems are better able to verify students’ enrollment and graduation status than are those agencies that collect aggregate data from schools and districts in more traditional formats. Additionally, some SEAs take a more active role in cleaning and processing these data, while others rely more heavily on their LEAs to clean these data points.

Top