Chapter 8—Needs Assessment: Defining “There”

Many states jump head first into developing or purchasing an LDS (or certain components) before spending time to think about exactly why they are doing so, what their stakeholders need, and what it will take to get the job done. Experience suggests that successful LDS development is roughly 80 percent planning and only about 20 percent building.* Careful planning can make the difference between a clumsy belly flop and a graceful dive into system development.

During the needs assessment phase, policymakers should establish the business justification for what kind of LDS they want and why. In addition, stakeholders should define their requirements so that the resulting system reflects their needs. Needs assessment is also another step in creating early buy-in for the project, getting everyone—both internal and external stakeholders who use, or would like to use, education data—involved in creating a vision for a system that they will one day find useful.

Why Do You Want an LDS?

Before the developers start defining requirements and building the system, some fundamental questions should be answered about why the system is being built and what it will do for the education community. Early on, decisionmakers should ask themselves the following questions:

	“Why do we want an LDS?”
	"What are our ultimate goals for the system?"
Preferably, the answers will not be:
	"To check off the 12 elements of the America COMPETES Act."
	"To keep up with other agencies."
Better answers are:
	"To improve instruction by helping teachers identify student needs, discover and practice the most effective teaching strategies, and tailor instruction; by helping administrators target teachers for professional development; and by helping researchers conduct more informative studies to identify effective strategies."
	"To inform policy and resource allocation decisions at the state and local levels with better information, and to help state program staff target district and school improvement needs."
	"To calculate academic growth; and follow students and maintain their academic histories as they advance to higher grade levels, transfer across districts within the state, drop out or transfer to private school or another state and come back into the system, and move into higher education."
	"To track staff and maintain their professional histories as they enter and progress through teacher preparation programs, receive professional development, and transfer among schools."
	"To streamline operations and improve data quality by automating processes such as data entry and loading, making data collection more efficient; by helping state staff produce federally- and state-mandated reports; and by conforming to broadly shared data standards."

What Do You Need from an LDS?

Once decisionmakers and other stakeholders are familiar with the LDS concept (see Book One: What is an LDS?), the benefits it can provide (see chapter 5 of Book One) what their current system looks like (see chapters 6, 7, and 9), and why they need such a system, they can create a vision of what they want their own LDS to do and what functionalities they want it to have.

Determine system requirements early so that expectations can be adjusted for everyone involved in the development process. Establishing goals and expectations for the system is also a good practice. This will help focus system design and keep IT staff, vendors, legislators, and others on the same page in terms of what needs to be done and what will actually be available in the future.

Everyone wants to improve student achievement, but the important question is, “How do you want to do it?” Policymakers and other stakeholders need to determinewhat functionalities they want and establish requirements for the developers. If system developers have clear requirements to fulfill, policymakers and other stakeholders are much more likely to get what they need.

Key questions for key audiences Ask decisionmakers: "What do you want and why do you want it?"     Ask education experts: "What will the system do for you?"     Ask system developers: "What will make the system work?"     Ask everyone: "How do we know if we achieved the goal?" "How do we identify success and anticipate failure?"

You may want a souped-up, cutting edge system that will cost a lot. Or you may just want a basic, no-frills economical system. Either way, you must be clear about what you need and what you can afford. Without careful reflection and planning, your organization may find itself with a poorly designed system that does not really do what it needs it to do.

Others' LDSs The National Center for Analysis of Longitudinal Data in Educational Research offers links to several LDS datasets along with descriptions of their source systems. The State Data page is a good place to research other agencies' LDSs.

Knowing what you do not know

It is sometimes difficult to figure out what you need because you may not be familiar with what you do not have. You may not realize that certain functionalities might make your job easier. To get started, you might compare your current system, as depicted in the self-assessment, to the ideal LDS outlined in chapter 3 of Book One: What is an LDS? Researching other data systems or visiting an education agency with a more sophisticated system can also provide context and insight into available options (see NCES's Personnel Exchange Network). Also, review the software applications on the market. Consider the functionalities they offer and determine if they would work in your environment and meet your stakeholders' needs.

When working with stakeholders, try to operate in concrete rather than in abstract terms. Instead of asking vague, open-ended questions about what they need, it may be helpful to start with questionnaires to gauge interest in certain LDS components and capabilities. For instance, an agency might survey staff and other stakeholders with a list of questions that the new system could be designed to answer. The staff could then rate the questions in terms of the value they think the answers would provide, thus allowing you to gauge stakeholder interest in a systematic fashion. And, since some stakeholders will inevitably be more outspoken in stating their needs in meetings and other venues, such surveys are a good way of leveling the playing field so prioritization of needs can be done fairly.

Specific questions included in such a survey might be drawn from the stakeholders themselves, and the rating process could provide a basis for prioritizing stakeholder needs. Surveys like this might be written for different groups. For instance, teachers might be asked to evaluate questions about curricula or student achievement, while other staff might be asked to assess operational functionalities such as data entry automation and data sharing between databases. Based on the answers provided and the resources available, a state might decide which data to collect, which tools and reports to offer, and which capabilities to prioritize.

Assessing what you need

Third-party needs assessment You may conduct your needs assessment on your own or work with a vendor to define your requirements. Both approaches are common. Some agencies hire a vendor to do the requirements gathering, allowing an expert to figure out what the education agency and its stakeholders need. If your agency is allowed to enter into personal service contracts, you may be able to employ a vendor under a time and materials contract to do the needs assessment for, or with, you before the RFP is written (see chapter 14) and the development project begins. While some agencies find this approach very helpful, others prefer to do the needs assessment, or at least a large share of this phase, on their own before the vendor is brought on board. In other words, they feel the agency should know what it wants, rather than let a vendor determine what it wants. However, if you elect to have a vendor do requirements gathering for, or with, you, make sure the contractor is knowledgeable about your agency and about education data. And make sure stakeholders are being asked the right questions to elicit helpful responses. Stakeholders usually know what they want, but they often do not know how to articulate their needs unless asked the right questions.

To help guide discussions, develop some questions that should be asked of the various audiences. Some examples are listed in table 3.

Documenting requirements in some form, such as in a needs statement, is very important. This is "a description of the functional needs, technical requirements, and security and ethical standards that need to be met by a technology solution" (National Forum on Education Statistics 1998). An agency should take its findings, categorize them, and try to boil them down to discrete needs. The resulting set of requirements can help guide the work of in-house system developers or help an agency find a product on the market that will meet its needs. If you are planning to hire a vendor (or several vendors), your findings will help in the creation of an RFP (see chapters 13–14). Either way, be sure your identified needs can be measured by tangible criteria, so that success in attaining the organization's goals can be assessed during evaluation of the system later on (see chapter 15).

After figuring out what you have and what you want, a "gap" analysis should be done to identify the discrepancies between "here" and "there," and determine what work will be necessary to achieve the desired system. For instance, if the agency does not currently have a student identifier system, but the needs assessment calls for one, a "gap" has been found that needs to be addressed.

Building an LDS with stakeholders' input is an iterative process. Give it time and be persistent. Expect needs assessment to be ongoing, rather than completed once at the beginning of the project. As parts of the system are developed and go live, continue to get feedback to find out what stakeholders think and what new needs they might have. New ideas often arise from new developments.

Realistic LDS expectations When negotiating with a vendor or working with in-house staff to design your LDS, consider asking for these important LDS features: Unique identifier system that allows individual student achievement to be tracked yet protects student privacy; Interoperability with other K–12 systems; Interoperability with PK and postsecondary systems; Teacher value-added data; Data warehouse (or comparable alternative); and Reporting and analysis tools with easy-to-use interface.

District difference: District-level LDS considerations Most of the discussions about LDSs focus on state-level systems. But state agencies are not the only ones building these systems—many school districts and regional agencies are as well. While fewer people may work on a local-level LDS, the same need exists and the same mistakes are often made. Below are questions and considerations for school-district staff to ponder before building their LDS. (Many of these are also applicable to a state-level effort.) When assessing the need for a district-level LDS, ask What are the existing options? Can the state offer LDS services? Can the region offer LDS services? Is it possible to form a partnership or cooperative with other school districts in order to share the LDS effort? Does your district have the resources to implement and maintain an LDS? Can you cover the initial costs? Will you need additional staff to maintain and manage data loads and reporting? Can you afford the additional costs of training staff to use the system? Can you afford to train staff to make informed decisions at the classroom-, school-, and districtlevel based upon the new LDS data? If the decision has been made that a local LDS is necessary, then Investigate what other districts in your state are doing, as well as the state agency. Ensure that whatever you do fits into the big picture and your investment will not be lost because the state agency requires use, or interoperability with, another unique system. Coordinate with the state agency to maximize alignment with state data standards. Consider using data standards such as SIF specifications to connect systems. Address quality issues at the source(s) of the data before they are integrated into the LDS. Ensure that the LDS is able to handle your district's unique local assessment data needs.

Top