Chapter 5—Data Quality From Bottom to Top

The best way to ensure the quality of data is to get them right in the first place, and to prioritize quality throughout the information life cycle. This approach relies on staff in the school as well as at the district and state levels. It starts at the source, typically in the school where teachers, clerks, and other personnel enter data. From the school, the data are sent to the district, where they are validated and/or audited; then to the state agency and federal government, where further quality assurance processes take place.

The Forum has more detailed information…
...about improving data quality: Forum data quality online courses Forum Curriculum for Improving Education Data: A Resource for Local Education Agencies (2007) Forum Guide to Building a Culture of Quality Data: A School and District Resource (2004)

In addition to the processes that check the flow of data "up" the ladder from the school, quality also relies on effective governance and communication back "down" again. Establishing effective data governance at the state and district levels provides a mechanism to help resolve problems and prevent finger-pointing or issue avoidance. Education agencies must move from ad hoc data management models to strategies that bring together all stakeholders from across the enterprise, create key governance groups, assign clearly defined roles and responsibilities, secure agency data, and ensure the data help achieve organizational goals. (For more information on data governance, see chapters 1, 2, and 3.)

State agencies and the federal government also establish policies, guidelines, standards, and reporting requirements that must be effectively communicated “down” to the data suppliers to enable successful and timely implementation at the local level. Likewise, school districts may create their own standards, guidelines, policies, and regulations to guide school data activities. They create data reporting calendars, data dictionaries, metadata systems, and business rules; assign responsibilities; and implement technology to facilitate data processes. These guidelines and procedures should be similar across program areas so that schools will have comparable experiences submitting various data. In other words, submitting data to one program area should not be very different from submitting to another.

Responsibility for data quality should ultimately rest with program area staff, rather than information technology staff. While IT can perform basic checks to see if the data "look" right, program area staff have a deeper understanding of the information and are better equipped to find errors. For instance, a data report might look right to IT if a total number of schools is generated, but a program area staffer may know if that number is actually correct. This in no way means that IT staff and the technology they manage are not critical factors in improving data quality. Technology that streamlines and automates data entry and sharing is indispensable to quality, as are validation procedures implemented through technology. However, to ensure data quality, education agencies often focus too much on technology and not enough on the data or the people and business processes regulating them. If reported data are inaccurate from the start, the best technology solutions will fail to transform them into quality data.

Data quality certification

The Kansas State Department of Education has been a leader in improving data quality at the local level. The state has created a professional development program that trains and certifies a range of school and district staff on data quality practices and techniques, as well as software applications. Click Kansas’s Data Quality Certification program for more information.

Provide Training and Professional Development

At the local level, where the life cycle of information begins, the data "creators"—the school teacher, counselor, nurse or secretary entering student data to the district or regional service agency; or vendor staff member building a report for the state agency—must be trained to ensure they produce high-quality data. This training should include best practices and procedures for creating and entering data; and the use of the technology employed to collect, edit, and report data. Staff should also be very familiar with all relevant policies, data standards, reporting requirements, and timelines.

Data appreciation leads to better data quality

Staff preparation should teach more than policies and procedures. Professional development programs, and ongoing communications throughout the enterprise, should help everyone understand why data are so important. Staff need to know how their handling of data affects the use of those data at all levels, shaping decisions from school funding to individual student learning. They must understand why the data are collected, how teachers and decisionmakers use the information, and how the work relates to the money the school or district receives. Understanding their uses will help staff appreciate why data must be accurate and timely, and provide an incentive to strictly adhere to procedures.

Data quality results from data use

If staff see data collection and reporting simply as chores to perform for an authority, they may not be sufficiently motivated to go the extra mile to ensure quality. To create an incentive to improve data quality, agencies must ensure data are used down to the school office and classroom levels. For instance, data will be more useful to practitioners if they have access to student-level data with reporting and analysis tools, or dashboards. District administrators can access the data to see how their schools compare with similar districts in the state. Teachers can view data in real time to inform lesson plans and tailor instructional strategies. Additionally, state agencies may turn the submitted data into useful reports for schools and districts perhaps enrollment lists or comparisons with other schools and districts. If data creators see that the data are used for high-stakes calculations to make their jobs easier, or to hold them accountable, they will have greater incentives to ensure the data are of high quality.

Data Auditing Procedures

The flow of data from schools and districts to the state LDS should include several safeguards to ensure quality. For example, on the way from the school secretary to the district and on to the state data system, certain procedures and mechanisms should be in place to check the data’s quality, identifying and resolving anomalies. Ideally, data should be checked for quality before they are loaded into the collecting systems. Some states use auditing mechanisms to check submitted data for problems, and validation reports to alert staff if they find any. Audits may include, but are not limited to, the application of business rules that

• compare data to prior year values to spot any significant changes where little or no change is expected (e.g., a change in a student’s race);
• identify invalid values (e.g., “Null” in a field that requires a numeric value, invalid codes, incomplete or blank fields, out-of-range or over-limit values);
• identify invalid formats (e.g., a date entered in incorrect format); and
• detect excessive use of certain codes (e.g., the frequent use of “Other”).

Such front-end validation procedures are preferable to back-end cleansing of data already loaded into the system. In addition, errors should ideally be corrected in the source files and resubmitted, rather than amended in the state or district’s system. If a correction must be made by state or district staff, a process should be in place to ensure that the source files are also corrected (Schutte et al. 2009). Errors may be identified as critical, which would require correction; or noncritical, which might require staff review, but not necessarily correction. Data suppliers should be notified of these discrepancies and be required to correct errors or confirm that any questionable data are, in fact, correct. Agencies must determine how often their data will be validated, and establish timelines for submissions and resubmissions. To further validate the information, some states also verify reports with district program area staff before the data are finally released to ensure that the numbers match districts’ expectations. This type of process helps guarantee the data reported are accurate representations of the reality.

Top