It is important to ensure confidentiality and protect student and staff privacy.
Chapter 2: Meeting the Challenges of Data Collection
As discussed in the preceding chapter, a safe environment, one that is free of threats and intimidation, provides a climate conducive to learning and teaching. Data collection is an important component in creating such an environment. Nonetheless, there are challenges to data collection. One challenge is accuracy-do the data represent what they are intended to represent? Other challenges include how interpretation of the data affects school image, especially how data are reported by the media, and costs of collecting data.
It is important to ensure confidentiality and protect student and staff privacy. It is necessary to exercise caution when sharing information with staff and other professionals serving schools. In addition, it is especially important to exercise caution when working with the public. The Federal Family Education Rights and Privacy Act (FERPA) prohibits schools and school districts from releasing most types of information about students. Many states also have legislation regarding the confidentiality of student records and information. The purpose of data collection is to support the mission of schools (i.e., to continue efforts to secure a safe and orderly learning place), and not to create a repressive environment.
Security is a more complicated issue now that many records are stored and transmitted electronically. Schools and school districts need to be sure that their computerized data are appropriately protected from unauthorized electronic access. Records can no longer be secured solely by locking the filing cabinet.
As previously described, incident data can be used to ensure the following:
To create safe and orderly places for learning, incident data must be accurate. Staff training and other efforts to ensure the reliability and validity of data are critical.
Chapter 4, "Collecting Data, " provides
specific details on the Task Force recommendations for the elements
to be included in an incident database. In addition to listing types
of incidents, variables are identified (e.g., time, locale) and defined.
A key feature of the incident database is the ability to interface with
student and staff databases.
It is important that the database records incidents beyond state-mandated or legal violations.
It is important that discipline referrals and disruptive incidents be recorded even when they do not cross the threshold for state-mandated reporting or legal violations. This information can be used to identify trends and link incidents to grade level, location, time of year, or other potentially useful factors (Cornell et al., 1999).
When designing a database system or purchasing one that is commercially available, users need to evaluate key features, such as whether users will be able to produce all reports necessary to identify problem areas. Problem reports and trend data can be used to set specific goals for preventing problem behavior, develop school improvement programs, and facilitate documentation of progress in addressing school needs (Cornell et al., 1999).
Training is a critical element of data collection. It is important to train all school staff on what kinds of problem behavior are to be reported to administrators. It is important for administrators to be clear on how incidents are defined, and for both administrators and data entry personnel to understand the specifics of reporting procedures. Cornell and colleagues (1999) advocate statewide training for all school divisions on the implementation of an incident-based data collection and reporting procedure, as well as training on the appropriate use of such information for school safety planning and procedures.
School level administrators will be better prepared to guide their
staff in district discipline policy if they themselves receive training
from qualified instructors. District level administrators can educate
school level administrators more effectively regarding state mandates
if they have received guidance on the subject. A coordinated, integrated
training program will provide both clarity of policy goals and consistency
Even with clear and concise definitions, training at all levels is crucial.
Providing guidance on discipline to administrators takes two forms: (1) training on recordkeeping and (2) instruction regarding definitions and classification of incidents and disciplinary actions. If school district administrators wish to compare incident data across schools, they need to do more than leave it to untrained school staff to simply choose the incident code that most nearly matches the infraction. Doing so may produce inconsistent data. Even with clear and concise definitions, training at all levels is crucial to ensure data comparability. Districts may want to consider forming discipline committees to review data, develop and implement plans for training, and revise codes as necessary. Including practice on identifying the types of misbehavior to be recorded would be beneficial. Examples of what does and does not meet the standard for each type of infraction will be important to differentiate among definitions (e.g., Battery (code 1300) versus Fighting (code 1700) versus Physical Altercation, Minor (code 2500) [See Chapter 4]).
Technical assistance staff in Florida provide guidance on incident reporting during regional workshops. School and district administrators are trained on how to categorize incidents and what level of problem behavior meets the threshold for state-mandated reporting. Before developing this program, state staff worked to identify reporting problems, revise and clarify definitions, and develop technical assistance materials such as video clips of examples and "nonexamples" of each type of incident. School and district administrators and law enforcement officials contributed to this process. The state also provides grants to school districts that need additional support to work with their schools on data collection issues.
Many districts in Florida follow this model (See Figure 2-1.). Training is provided to district staff, school administrators, and data clerks. This training addresses:
For example, "disorderly conduct" that rose to the level of "state reportable" was a significant problem one year in one district. District staff felt that school staff were coding many incidents as "disorderly conduct" when a less serious code would have been appropriate. Training focused on the definition of "disorderly conduct, " and the number of reported incidents decreased substantially. Another district found that some principals were recording fights as multiple incidents-one incident for each participant. They held training on the structure of their database, which has separate modules for incident and perpetrator information; thus, the incident itself is recorded in one data file and the information on each perpetrator is recorded in a separate data file.
Documentation within schools is most likely to be accurate and consistent if school administrators or their assistants code infractions, but it is still very important that all school staff are trained regarding prohibited behaviors. Training within schools could follow the state and district model provided in figure 2-1.
School staff members need to be trained on what to report. Training will reduce the likelihood of inconsistent reporting. In addition, training can help ensure that minor problem behaviors are reported before they escalate. Including information on incident definitions and reporting procedures in teacher handbooks is recommended. Efforts to ensure consistency among staff members in the level of misbehavior that will be tolerated are also important. In addition, regular review of incident reporting and discipline problems at faculty meetings is useful.
2.1.3 Ensuring Reliability and Validity
Reliability and validity are important components of data collection. The following components ensure that reporting is consistent (reliable) and that variables measure what they were intended to (valid):
Defining problem behavior. Chapter 4 describes the types of incidents to track and elements to record. The list of incidents in Chapter 4 is extensive; however, not all incidents will be appropriate for all schools and school districts. In addition, the list may miss incidents important to particular jurisdictions or schools.
Decisions on which incidents to include need to be made by the people
they affect. Involving stakeholders in the decisionmaking process will
generate support for data collection efforts-no effort can succeed without
the support of front-line individuals. If data collection decisions
are being made at the school level, it is critical to include teachers
and other staff on the team that designs the incident-reporting system.
If the decisions are being made at the district level, it is necessary
to include school administrators and school staff in the process. If
decisions are being made at the state level, it is important to include
school and school district administrators in the process. It may also
be helpful to include parents and community members on the incident
committee. Information that is useful to individual schools may not
be useful to the school district as a whole. For example, it may be
important for a school to know whether an incident occurred in the locker
room or in the cafeteria, but the school district may not need to know
this. Similarly, a state may not need to know whether an attendance
policy violation consisted of skipping class or forging an absence note.
Communication is essential to ensure reliability and validity.
Communication is essential to ensure reliability and validity. It is critical to clearly describe behaviors that have been defined as infractions in student, staff, and parent handbooks. Students need to know what behaviors are unacceptable before they can be expected to behave appropriately.3 Staff must know the behaviors for which they are expected to discipline students before they can do so.4 Finally, parents must be informed about how their children are expected to behave.
Data checks. It is crucial that data be reviewed regularly
for completeness and accuracy. It is also necessary to verify that all
data elements are being recorded for each incident. During the data
review process it is important to examine incident elements being tracked
(e.g., time, perpetrator).
Assigning a key person the responsibility for oversight of all aspects of incident reporting can improve accuracy.
At the school level, all incidents recorded on paper need to be entered into the incident database. Staff need to note on each paper reporting form that the incident has been recorded before the form is filed. It is important to establish a system for recording this information (i.e., who enters what information when) to increase the likelihood that information will be recorded as intended. Reviewing paper files to determine whether or not incidents were recorded appropriately may also be useful in managing data entry systems. Assigning a key person the responsibility for oversight of all aspects of incident reporting can improve accuracy.
One way to assess accuracy is to compare a sample of incident forms to what has been recorded in the database for those incidents. When assessing accuracy, it is also important to consider differences from month to month or week to week in the number of each type of incident. If significantly more or fewer "bullying" incidents occurred in one time period than another, it is important to investigate why that is so. If a new bullying prevention program has been implemented, it would be logical for the number of incidents to increase just after the program takes hold. At that time staff will be more sensitive to the kinds of behavior that are unacceptable, and for incidents to decrease after a program has been in place for some time, as students learn more positive ways of interacting. The data must also "make sense." In other words, did a school really have 10 battery incidents last month, or is someone miscoding another type of infraction? Other basic data checks include assessing what percentage of students committed rule violations and which violations are the most frequent. An effective database has built-in checks for consistency of data. For example, if a weapon-related incident is recorded, the computer program will automatically prompt for type of weapon.
When analyzing schools within a district, district staff need to ask whether it is reasonable that School A has more "fights" than School B, given differences in enrollment. If not, more training may be required. Similarly, it will be useful for district and state staff to ask whether it is reasonable that District A has the same number of "fights" as District B, given differences in enrollment.
Compare and contrast. It is useful to compare incident data to other sources. Serious infractions (e.g., illegal behaviors) can be compared to police reports. Incidents involving injury can be compared to hospital records. Levels of problem behavior identified in self-reported data from student and teacher surveys can be contrasted with the level of problem behavior being recorded in incident databases. For many reasons, rates reported from these sources will differ from school or district databases. However, it is important to investigate significant disparities. For example, if students are reporting very high rates of drug use but there are very few drug infractions recorded, it will be valuable to learn why the difference exists.
Problem behavior will not go away if ignored.
It is reasonable to be concerned about how data will be used. Such concerns should not become barriers to effective data collection. Problem behavior will not go away if ignored. Without information on what kinds of problem behavior exist, schools will not be able to take steps to address their problems. It is important that information lead to action, not frustration. One use of data is to identify students, schools, and school districts in need of additional supports. Schools cannot demonstrate improvement without a way to establish a starting point-an incident database can provide this.
Parents and the public want schools to produce well-educated students.
Given this focus, it is important to educate the public on the need
to invest energy in creating safe and orderly schools. The premise that
an orderly environment is an important factor in student achievement, and that students cannot be successfully engaged in academic work in
a disorderly environment, must be communicated to parents, taxpayers, and policymakers (Squires, Huitt, and Segars, 1983). One way to do this
is to develop strong media relations.
Approach problems as "needs"to let the public know what is needed to accomplish specific goals.
Positive media relations. Working with the media can help schools, districts, and states present data in ways that will help strengthen public support for their data collection and prevention efforts. In "A Short Guide to School Public Relations, " Kinder (2000) explains that when the facts are known, the vast majority of people are supportive of the efforts of their school leaders. He notes that the time to launch a public relations effort is not during a crisis. A well-designed public relations program aims to prevent problems, not to solve them. Sound public relations efforts can be used to build public support for schools (Muir, 1999). The National School Public Relations Association recommends enlisting the media to publicize efforts to promote school safety (Price, 1999). Another important strategy is to ensure that all communication contains some form of success story, even when problems are being identified, because this conveys the message that something can be done to improve matters (California Department of Education, 1997). Other experts on school public relations recommend that, when school officials are talking with the public, they approach problems as "needs"-that is, to let the public know what is needed to accomplish specific goals, and what the community can expect to gain for its support (Muir, 1999; Gaskins, 1999). If the school needs money to implement prevention programs, the public must know how effective these efforts can be. For example, Patterson Elementary School used the Effective Behavior Support Program to halve the school's office referrals. The program is aimed at teaching students appropriate conduct and involved tracking the number of students referred to the office, for what offenses, where the violations took place, and concentrated on the most disruptive students. (Sack, 1999). Such success can be very persuasive when asking the public's help in raising the money to implement a program in a specific set of schools.
Public perception. Before sharing data with people outside the school building, school staff need to be aware of this information. Communicating with school staff is a vital part of safety planning and discipline improvement, and staff members will particularly appreciate receiving timely information directly from administrators. Providing staff members with data on behavior improvement gives them an opportunity to be proud of their accomplishments. In addition, it is important that they know what problems need to be addressed and that they are involved in deciding how the problems will be addressed (Bernhardt, 1998; Ross, 1999). Data can easily be shared at faculty meetings or through school improvement teams. Next, share information with parents. Presentations to parent/teacher associations and school newsletters are two possible communication tools. Reporting mechanisms and presentations are described in Chapter 4.
The California Department of Education (2001) suggests that, when reporting to the public, administrators stress the positive messages illustrated by the data (i.e., safety instead of crime) and how data have been used to develop successful programs. When administrators communicate statistical data in such a positive way, the public is presented with a solvable problem, not an insurmountable dilemma. Presenting data offers an opportunity to form a partnership with law enforcement or other community agencies-another potential resource for schools.
Implementing a system that allows for early identification and intervention could result in substantial cost savings.
It is likely that there will be costs associated with efforts to clarify definitions; aligning handbooks, rules, policies, and practices; and training staff in what and how to report. However, these initial investments can have great payoffs. One of the most important results would be fewer incidents of problem behavior and a net gain in time spent on the task of learning. There is a strong association between students having academic and behavioral problems (e.g., Hinshaw, 1992; Yoshikawa, 1995). High levels of problem behavior are associated with early grade retention and grade retention is costly, adding at least the expense of an additional year of school to a child's education (Byrd and Weitzman, 1994). According to the National Center for Education Statistics (2001), per-pupil spending averaged nearly $6,500 nationally for the 1998-99 school year. Implementing a system that allows for early identification of students with serious behavior problems and for early intervention (thereby supporting the prevention of grade retention and remediation of other problems) could result in substantial cost savings. The implementation of a system that assists with the prevention of violence and vandalism will also result in cost savings. In addition, schools with low rates of violence and vandalism are more effective places where more learning occurs (Squires, Huitt, and Segars, 1983). Thus, the rationale for data collection meets educational goals of a safe climate for academic achievement.