The Statewide Longitudinal Data Systems Grant Program (SLDS) hosted its second
Annual Fall Grantee Meeting on November 28-29, 2007, in Arlington, VA. The Meeting
served as a forum for dialogue, collaboration, and the sharing of best practices,
providing the opportunity for more than 70 representatives from the twenty seven
FY2006 and FY2007 grantee states to share solutions and ideas with one another
and to take home information on topics they identified as critical to their
projects in the upcoming year.
| Session I |
Approaches to Longitudinal Data Systems |
| |
Presenters:
John Calderone, Wisconsin Department of Public Instruction
Andre Smith, Florida Department of Education
Baron Rodriguez, Oregon Department of Education
Moderator:
Joe Dietzel, North Carolina Department of Education
|
| |
Wisconsin, Florida, and Oregon presented their approaches to Longitudinal Data Systems. The discussion
focused on the need for Data Quality and the issue of staffing costs; the cost of hardware and software
transitions; the necessity of buying enterprise state licenses; managing future data requests as systems
are built; and the importance of using Business Intelligence models to manage access.
- Wisconsin is building its own system, enabling the state to design it to fit its needs, but specifically for the state leader-the system includes a "school report" system that allows one to print all necessary information for the State Superintendent when conducting school visits.
- Wisconsin stressed the importance of professional development, in order to enable educators using the data to improve student achievement.
- Wisconsin was excited about the final product of its EDEN portal.
- Florida and Oregon addressed the need for the system to connect between districts and/or with other agencies in the state.
- Florida encouraged the required support from policy makers and decision makers in order to create a sustainable model.
- Oregon stressed the need to be able to transfer records electronically, and discussed its method for engaging stakeholders and assessing their needs.
- Oregon took a regional approach to its LDS development, setting up 6-7 regional data collectors to collect information above and beyond what the state requires.
- Data Quality can be improved by increasing transparency, addressing entry problems, and implementing checks.
Download Zipped PowerPoint Presentations:
Florida's PK20 Education Data Systems (163 KB)
Oregon DATA Project (Direct Access To Achievement) (754 KB)
Approaches to Longitudinal Data Systems: Wisconsin's Game Plan (3.97 MB)
Top |
| Session II |
U.S. Department of Education Updates |
| |
Presenters:
Allen Ruby, National Center for Education Research (NCER)
Amy Feldman, National Center for Education Evaluation and Regional Assistance (NCEE)
Peter Tice and Jeff Owings, National Center for Education Statistics (NCES) |
| |
Participants updated attendees about a variety of resources available from different agencies within the Department of Education: The Institute of Education Sciences (IES), The Regional Educational Laboratory Program (REL), and the National Center for Education Statistics (NCES) Longitudinal Studies Programs. The discussion led to questions of the participants and their specific areas of expertise. They centered on the links between studies, consideration of pre-school data including disabled students, and FERPA issues.
- Runs a wide variety of grant programs aimed at improving education outcomes of students. States can receive help in preparing grant program applications from the officer for a particular grant program (http://ies.ed.gov/ncer/funding/); for example:
- Publishes guides that rate effectiveness of programs and curricula:
- Also publishes policy-focused research papers based on statewide data systems, such as how teacher policies affect recruitment, student outcomes, etc:
- Consists of a network of ten laboratories that serve the educational needs of a designated region by providing access to high quality scientifically valid education research through applied research and development projects, studies, and other related technical assistance activities (http://ies.ed.gov/ncee/edlabs/).
- Releases publications from Fast Response Projects and Rigorous Studies, which have gone through a strict peer review process (http://ies.ed.gov/ncee/edlabs/projects/index.asp and http://ies.ed.gov/ncee/edlabs/projects/projectarea.asp?pid=2).
- SEAs and LEAs can request technical assistance from their REL through short term fast response projects, which interpret evidence-based education research, and/or through rigorous experimental studies that evaluate interventions.
- Develop survey and assessment instruments that may be useful to states.
- Allow for longitudinal analyses relating a variety of factors to student educational outcomes.
- Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K:11)
- A new kindergarten cohort study will begin in the fall of 2010 and follow this group through the spring of fifth grade in 2016. The study will examine students' school readiness and the relationship between their kindergarten experience and later school performance, with a focus on growth or change in cognitive and non-cognitive domains. It will also collect data useful for research into the interactions of school, family, and community with student achievement and development.
- http://nces.ed.gov/surveys/SurveyGroups.asp?Group=3
- High School Longitudinal Study (HSLS: 09)
- HSLS:09 will begin with a cohort of ninth graders in the fall of 2010 and follow this group through high school and postsecondary education. The study will examine factors affecting students' decision making regarding high school coursework in STEM courses and postsecondary choices. It will also examine the role of the guidance counselor in assigning students to specific courses. A major emphasis of this study will be centered on math and science.
- http://nces.ed.gov/surveys/SurveyGroups.asp?Group=1 (Coming Soon)
Download Zipped PowerPoint Presentations:
Forum Overview (522 KB)
Regional Education Laboratories (RELS) (740 KB)
High School Longitudinal Study (HSLS) (145 KB)
Early Childhood Longitudinal Study (ECLS) (94 KB)
Top |
| Session III |
NCES Handbooks Customization Tool Demonstration |
| |
Presenter:
Duc Ta, Kforce Government Solutions |
| |
The NCES Handbooks Online provide guidance on consistency in data definitions and maintenance
of education data. In an effort to encourage more states to use the handbooks, NCES has developed a state
customization tool. State personnel will be able to use the customization tool to build a data dictionary
by adding to, deleting from, and editing the NCES data elements and option sets. The tool offers the
advantages of a built-in foundation of data elements and option sets; state control of the content
update schedule; and a well-defined database hierarchy.
- Deborah Newby, CCSSO, explained how CCSSO manages the handbooks and placed them online.
- Duc Ta, Kforce, presented the online customization tool's Administrator Console, available at: http://nces.ed.gov/programs/handbook/
Download Zipped PowerPoint Presentation:
NCES Handbooks Online Presentation (52 KB)
Top |
| Session IV |
Portals and BI Tools |
| |
Presenters:
Neal Gibson, Arkansas Department of Education
Jeff Sellers, Florida Department of Education
Bethann Canada, Virginia Department of Education
John Brandt, Utah State Office of Education
Moderator:
Bob Hackworth, Kentucky Department of Education |
| |
The discussion centered on gathering good data, preparing reports and usability for all stakeholders.
- Data quality is important, but access is even more important and should be geared towards teachers.
- Participants learned that it is better to start by providing simple data first and slowly feeding more information to the audience.
- Access to Virginia's portal is role-based (data management and decision support tools) and is granted locally according to local policy.
- Without usage, there is no point in having a system (keep in mind that what makes sense for developers does not necessarily make sense to teachers and principals).
- Vendor provided Virginia with a CD demo to use for presentations-this has been extremely useful in achieving buy-in from stakeholders since it demonstrates the granularity that VA data go to and how data can help teachers help students achieve (e.g. data help a teacher see those students on the cusp of passing the assessment test and enable them to work with them to achieve passing scores).
- Arizona believes that the state should give valuable information back to the local level, so that local stakeholders see that the information they receive from the state level can be of use to them, which in turn helps the state convince local levels to share control of the data.
Download Zipped PowerPoint Presentations:
BI Problems (1.3 MB)
Sunshine Connections (1.6 MB)
Virginia's Educational Information Management
System. Decision Support Tools (774 KB)
Top |
| Session V |
Data Quality |
| |
Presenters:
Kathy Gosa, Kansas Department of Education
Bob Beecham, Nebraska Department of Education
Leon Nelson, South Carolina Department of Education
Moderator:
Corey Chatis, Tennessee Department of Education |
| |
Data entry personnel should be trained, motivated, recognized and rewarded for high quality data input.
Accuracy is critical--especially to funding issues. Clarification of definitions of all data elements and
data governance affects all program areas.
- In Kansas, Nebraska, and South Carolina efforts to improve Data Quality include the implementation of a Data Quality Certification Program at the local level; forming Education Data Quality Partnerships within the state; recognizing and rewarding those who are entering data correctly; documenting definitions of data elements and communicating this metadata documentation to the local level; and putting governance structures in place that facilitate communication and accountability.
- Since Data Quality "begins at the source" (e.g. the individual who enters the data), providing ample and appropriate training at the local level, and recognizing and rewarding personnel for accuracy in data input, are critical.
- Automated data audit processes also work towards improving Data Quality.
Download Zipped PowerPoint Presentations:
Kansas Data Quality Initiatives (14 KB)
SLDS on Data Quality: Presentation and Discussion (45 KB)
South Carolina: Achieving Quality Data (575 KB) Top |
| Session VI |
Security, Data Access |
| |
Presenters:
Don Houde, Arizona Department of Education
Mark Vocca and Dave Williams, Connecticut Department of Education
Bethann Canada, Virginia Department of Education |
| |
When developing requests for proposals, make sure to think about security up-front and include it.
Security issues are important, time consuming and expensive.
- Arizona has adopted a Sarbanes/Oxley-like model for security accountability.
- Attendees were advised to have employees and contractors sign non-disclosure forms and researchers to sign MOU.
- In addition to FERPA, Virginia uses the "Privacy Act" to prevent data from being shared across state agencies.
- Automatic encryption for hard drives; standardized data encryption across the whole state; thumb drives are encrypted and require passwords to access the information; difficult to compromise.
- Mandatory security training for all employees, to be renewed every year, is critical.
- Connecticut raised the issue of the difference between authentication and authorization and building security.
- It is important to try to be transparent and communicate system to the outside and even governance structure/culture inside so that they can see that the information they have is not so top secret.
- Presenters advocated moving towards a model where no one group has complete control of system-two sides, a governance side and a technical side.
- Virginia's statewide program on security has a review process yearly and is audited every 3 years.
- North Carolina suggested an excellent presentation on this called "Identity 2.0" by Kate Greene published by MIT Review (http://www.technologyreview.com/Infotech/16509/page1/).
Download Zipped PowerPoint Presentation:
Arizona: ADE Data Security Initiatives (521 KB)
Top |
| Session VII |
Data Records Exchange |
| |
Facilitator:
John Brandt, Utah State Office of Education |
| |
States are trying to address the need to transfer data and records to LEAs, and to address the need to
transfer data across state lines.
- South Carolina and Nebraska want to develop transcript and record exchange to work with higher education.
- Indiana uses Docufile for transcripts and is looking at record exchange between LEAs. They emphasize the need to distinguish between storing data and moving data across sectors (LEA - LEA, LEA - SEA, and LEA - Higher Ed).
- Maine would like to pursue sharing data across state lines, recognizing the challenges involved.
- South Carolina follows the model that if it builds a system that looks appealing to other departments, they will in turn see the benefit and possibly fund the integratability from their own budgets.
- Participants stressed the need to distinguish between e-transcripts, student record exchanges, storage and facilitating transfer.
- Participants also discussed the need to be politically savvy in terms of selling the system to policy makers.
Top |
| Session VIII |
User Training |
| |
Presenters:
Corey Chatis, Tennessee Department of Education
Lynda White, Maryland Department of Education
Baron Rodriguez, Oregon Department of Education |
| |
The cost of training is expensive. However, states suggested that there may be enough commonalities in
training related to the use of education data (separate from state-specific analysis tools) that states can
collaborate and share with one another.
- To customize instruction for different stakeholder groups (e.g. data managers, district staff, and instructors), Oregon conducted needs assessments-through facilitated sessions and written surveys-to identify the specific data and training needs of each user group.
- Tennessee has begun to pilot its training with districts. Training covers content of the data warehouse, data elements within each subject area, and what can be done with the data. The training also reviews reports created thus far, and gathers feedback for additional reports that LEAs may need. The state will eventually create a "power group" of core users at the district level.
- In partnership with Johns Hopkins University, Maryland is working to develop, and make available online, professional development for school, classroom and district level use of data.
- Training for stakeholders, particularly teachers, can be broken down into 2 phases: using statistics in education, and using the available analysis tools.
- Web trainings with visuals and audio (available via links on the portal) can be provided to large audiences and possibly at a lower cost than in-person trainings.
- Requiring affidavits of non-disclosure and providing different levels of training (which determine different levels of access to data/reports) are ways that states can protect themselves from inappropriate use of data by stakeholders.
Top |
| Session IX |
Longitudinal and Stakeholder Analyses Panel |
| |
Presenters:
Margaret Ropp, Michigan Department of Education
Kathy Gosa, Kansas Department of Education
Jeff Sellers, Florida Department of Education |
| |
The use of the matrix tool and its impact raises many questions. The participants raised the possibility of
more focused group to address the issue.
- Kansas uses the evaluation tool matrix by asking many questions about its impact on data, policy and evaluation.
- Initial tool to help states:
- evaluate requests for data ;
- prioritize requests for stakeholders; as well as
- additional ideas and questions.
- Two decision points are needed in designing stakeholder analyses: data for compliance and data for decision making.
- Participants discussed the need to find additional ways to apply longitudinal data.
- Michigan is moving from collecting aggregate data to individual student data.
- Florida demonstrated how they are using data to track teacher pipeline and retention issues and presented their findings as a demonstration of capabilities for their analyses.
Download Zipped PowerPoint Presentations:
FL Teacher Data (441 KB)
Longitudinal and Stakeholder Analyses (396 KB)
Teacher Analyses/Reports Pilot Project Working Meeting (283 KB)
Prioritizing LDS Reports for Stakeholders (307 KB)
Top |
| Session X |
FERPA Discussion |
|
Facilitator:
Baron Rodriguez, Oregon Department of Education |
|
What data are necessary for teachers to see regarding their students? Is there an impact in gaining this
knowledge?
- Approaches with regards to free and reduced price lunch information and immunization records: Participants discussed issues related to providing such data to schools and teachers in relation to identifying students at risk.
- Clarifications of FERPA regulations on the topic of re-disclosure were submitted to OMB for review and eventual publication for public comment. They should provide more clarity on sharing data between agencies and LEAs.
- State legislative language can significantly impact the ability to create p-16 systems.
Top |
| Session XI |
Interoperability |
|
Presenters:
Anne Brinson, Indiana Department of Education
Tom Olson, South Carolina Department of Education
Rick Rozzelle, Tennessee Department of Education |
|
This discussion of interoperability focused on the Schools Interoperability Framework (SIF).
- Participants stressed the need to test vertical integration to improve data collection process from the districts not on state hosted SIS; to install much stronger data validation system; and to pull data on demand.
- Some participants had not been successful and questioned whether SIF was the right choice for them.
- South Carolina expanded to full SIF license, so districts could communicate horizontally to include all school districts; districts bought own agents-mostly for health, library, and management systems.
- Use website to post queries and to ask for submissions of data, but will be replacing with SIF based system making such queries and submissions automated so that the State can get data when they need it (as frequently as every night if they wanted), allowing the State to provide Districts with errors immediately, which speeds up the whole process.
- Virginia encouraged states to get involved if they want their voice heard by the vendors and stated that there is opportunity to do this through membership. They also said that the education industry has an obligation to band together and get involved in the standardization/SIF discourse.
Download Zipped PowerPoint Presentation:
TNDOE Next Generation Data Collection/Integration Tools (118 KB)
Top |
| Session XII |
EDFacts |
|
Facilitator:
Shawn Franklin, Nevada Department of Education |
|
Ross Santy and Pat Sherrill, U.S. Department of Education, were present during this session to hear
from the states and answer questions. Some states find EDFacts invaluable, but others burdensome.
The new EDFacts online metadata tool should address their concerns.
- How can states create EDFacts reports that would have value for school districts?
- States collect more information into SLDS. EDFacts is only a subset of the information needed by SEAs to evaluate impacts on student learning. Comprehensive data quality tools need to be in place for the data to be reliable and therefore useful and not harmful.
- EDFacts is actively working on an online metadata tool which should be released in February, 2008. It will list how state category sets match to EDFacts. Could be used to footnote tables. Some would be mandated to use but they can create some optional tools to help states.
Top |
| Session XIII |
Evaluation Procedures |
| |
Presenters:
Lynda White, Maryland Department of Education
Neal Gibson, Arkansas Department of Education
Nancy Haefeli, Ohio Department of Education |
|
Developing ongoing, formative, and summative evaluation procedures for determining whether the longitudinal
data system meets the needs of key stakeholders, as well as for assessing the system's effect on student learning,
are requirements of the SLDS Grant Program.
- Representatives from Arkansas, Maryland, and Ohio presented their methods and procedures for addressing these requirements: e.g. conducting a random sample study of student achievement before and after the system is implemented; monitoring of system's usage over time; and surveying of system users for feedback.
- Arkansas and Ohio are using outside vendors (Metis Associates and Hezel Associates, respectively) to survey system users.
- When gathering feedback from users, evaluations/surveys should have a limited number of questions and/or response time (e.g. Arkansas' goal is for 7 minutes or less), in order to encourage maximum feedback.
|