Whether an agency is building a data warehouse or adding reporting and analysis tools to an existing depository, someone has to develop and implement these products. Whether to take on these tasks within the agency or hire a vendor to tackle them for, or with, you is an early and crucial decision. This chapter's goal is to help you become an informed builder or consumer before you begin construction or hire vendors, in order to ensure you end up with a system that suits your needs and is worth the money. A collection of lessons learned and good practices, this section will help you choose, and steer you in a good direction once you have decided to either build or buy a solution, or perhaps pursue a combination of the two.
For some states, the choice between building or buying is obvious from the outset. For others, much deliberation is necessary. The decision depends on many factors. For instance, organizational characteristics such as technical capabilities, workload capacity, and culture are all important considerations. Does the organization have, and will it continue to have, the staff and expertise needed to carry out the project and maintain it throughout its life cycle; or is additional staff or a vendor needed? Beyond inhouse characteristics, organizational preferences play a role, deciding, for example, the fundamental question of where responsibility will reside—in-house or with a vendor? Your organization should also consider other questions:
|What can we build in-house that we cannot get, or do not want to get, from a vendor?|
|Can we get what we want from a vendor?|
|Will we need to settle for a system that does not completely meet our needs? Or, will we need to spend resources to modify the product to meet our needs?|
|Do we have the technical capability to build the system on our own and keep up with evolving data standards (e.g., federal reporting requirements)? If so, can we do a better job than a vendor?|
|Can we complete the task more quickly than a vendor? Or, can we meet deadlines without external assistance?|
|What if we lose key staff during the project?|
|Can we reduce project risk with a vendor?|
|Can we save money by building on our own or will it be cheaper to buy? (Consider ongoing maintenance costs for a vendor-built system and the savings you might have if knowledge resides in-house.)|
|What will be the ultimate trade-off between relying on a vendor and developing the expertise in-house?|
|What procurement approaches have other states taken?|
Some states prefer to keep the job in-house. They may have great staff with the necessary expertise and extra resources available to direct those staff members to the new project, or the ability to hire additional staff to do the work. Some states believe that, if you can afford to do it, build the LDS in-house. This, they say, is ultimately more cost effective because add-ons to vendor-built systems can cost a lot of money. If you have the capability, keeping the knowledge and ownership "in the family" will free you from the potentially costly prospect of having to rely on a vendor for ongoing services. Some states also feel that, in general, it takes just as much effort to work with vendors to make sure specifications are met as it does to build applications in-house. And working with staff may help ensure long-term sustainability because important knowledge about the data, source systems, reporting needs, making modifications, and other key issues are already in-house. And in fact, some research findings suggest that if an agency's capabilities are strong, or even just moderate, building is the better option: the organization should use its resources to tackle the project on its own and to further develop its staff expertise rather than bring in outside help. The staff's greater understanding of the organization's culture and goals may, in the end, compensate for a less technical skill set (Nevo et al. 2007).
Other states see hiring a vendor as the best way to go. This may be because they like a product on the market and see no need to "reinvent the wheel" when it has been used elsewhere with satisfactory results. States often purchase such off-the-shelf solutions and "brand" them—that is, putting their education agency's logo on a store-bought solution. By using a vendor, the state is freed from the details and can focus on the big picture and on selling the system to stakeholders.
Another reason to use a vendor is that an agency with limited resources and capabilities may find doing the work in-house to be unrealistic, even if it wants to. If the state does not have a staff member with the same level of knowledge as a vendor's project manager, or the capacity to build its desired product on its own, it may need outside assistance. In fact, research suggests that when internal IT capabilities are weak, the support of a knowledgeable, external consultant can boost productivity and reduce overall costs (Nevo et al. 2007). In addition, state hiring regulations that limit the size of SEA staff or impose other constraints may also make buying a more attractive idea. These rules can make it difficult to hire programmers. In this case, or if the organization is having trouble finding or keeping the necessary in-house staff, hiring vendors may be the simpler option.
A short timeline might also necessitate the use of vendors as the number of programmers, architects, business analysts, project managers, senior developers, and other staff they have available may allow the project to move much more quickly than it could if shouldered by in-house personnel. Provided your organization can find a vendor(s) and product(s) that will meet your stakeholders' needs, a commercial, off-the-shelf solution may also offer economies of scale in terms of leveraging design, including scope of data elements and requirements, warehouse structure, data model, data dictionary, submission procedures, etc.; as well as with development, testing, enhancement, support, optimization, documentation, and methodology across many customers. Whatever the reason, done correctly, choosing to bring in a vendor can be a very rewarding and successful course of action. Of course, every state's experiences will be different, and careful self- and needs assessments are vital whether you build or buy.
If your organization has decided it can handle the project without the assistance of a vendor, either the necessary staff are on board or new staff members will be hired. A central challenge your organization may face is finding and holding on to skilled technical staff. These experts are often drawn away from education institutions to the private sector for reasons such as higher pay. Many states have lost key team members in this way and such losses caused major setbacks to their project. Therefore, before you decide to build a solution, ask your organization how it will safeguard against this loss of staff and the knowledge they hold. If staffing is an issue, going with a vendor may be a more secure option.
If you have decided not to go it alone, your organization has chosen to buy a product and/or services from a vendor. You must now figure out what you need, what is available, what kind of relationship you want with your vendor, and how much you are willing to spend. Then you will need to write a good request for proposals (see chapter 14), interview some bidders, and select a vendor (or consortium of vendors). Finally, you must forge an effective relationship with the chosen vendor to get the product you require.
Many agencies use outside contractors and consultants to build their LDS or various components of the systems. These agencies may have one in-house employee who is the project manager, while consultants serve as programmers and system builders. Other states hire project managers as part of the vendor team. State agencies without the necessary expertise rely more heavily on their vendors and often seek a consulting firm they will not need to micromanage. However, states should be somewhat wary of depending too much on vendors, since they could unexpectedly close their doors or scale back their services.
States with considerable in-house knowledge may seek a relationship that gives them significant control over the vendor's work. These states act as CEOs overseeing the work of others. In this case, vendors may be hired to do the technical and programmatic work, and may even participate in committees to bridge the divide between their staff and the in-house executive levels.
Once you have written and sent out the request for proposals (see chapter 14), bids and proposals will come in from competing vendors. Now, of course, you have to decide which vendor to hire. You should use the proposals to narrow the field to a small set of top contenders. Keep in mind that vendors vary in many ways, including in their bidding practices.
In making your decision, look beyond the vendors' marketing to the details of what they can actually provide. Two core principles to follow are: know the product, and know the team. Rather than taking the lowest bidder, be sure the vendor can actually fulfill the contract, at the agreed-upon price. Look at the vendors' references, prior work, and available staff to try to evaluate them. Consider their experience working with states similar to yours in terms of size and student population. Will their solution accommodate your specific needs? Do they have enough staff to deal with possible turnover? Do they have the staff to adequately service all of their contracts, oversee the work, and complete quality deliverables on time? Or are they stretched too thin? In the end, careful selection may save you the costs of hiring a second vendor later on to complete the work or fix issues left by your original consultant. Also, as you will be spending a lot of time with this team, social considerations are also important. Will you like working with the project manager and staff?
Many agencies hold face-to-face interviews with a set of prospective vendors. These meetings offer the advantage of allowing you to meet the team of people with whom you will actually be working. Consulting firms should give presentations, demonstrate their products, field questions, and be allowed a chance to ask some questions of their own. A review board might be created by the education agency to evaluate the various vendors and should include agency as well as local leaders and prospective users of the system to get a variety of perspectives and insight. Other agencies do not hold face-to-face vendor interviews. They make do with reference checks, answers to standardized questionnaires, and phone and online meetings. At the very least, they say you can learn a good deal about a vendor by the types of questions they ask. What is their level of technological knowledge and capability? How familiar are they with education data and their nuances? For instance, a vendor might have an extensive background in finance data and might give you a product that would work well for a financial institution, but not so well for an education agency. Make sure the vendor is ready for the task at hand.
Create a Wiki
One state is in the planning phase of developing a wiki, a "piece of server software that allows users to freely create and edit web page content using any web browser."* This resource will be used to create a more collaborative environment in which the state and its vendors can work. The wiki will allow the state to post documents and receive feedback from vendors. After a substantial amount of information is accumulated, the wiki may eventually evolve into a user guide on the LDS.
Do not expect a vendor to hold your hand and guide you through the project design process. While many vendors have extensive experience supporting education agencies, others are not experts in education data or needs. In this case, many decisions will be left up to the client organization—you. A state or district must therefore have a complete understanding of what is planned in order to ensure their needs are met. Detailed planning must be done upfront and cannot be avoided. At the same time, the vendor team should have some knowledge of the education agency's data, or be willing to learn about the nuances of education data (for example, the importance of striking the right balance between confidentiality and not eliminating too many dropouts from the dataset). If necessary, ensure you have the time commitment from the in-house project manager, as well as the subject area experts and IT staff, to educate the vendor on source systems, reporting needs, validation of work, and other important issues.
Make sure the vendor staff working on your project are up to par. Some vendors will give you a top-of-the-line project manager, for instance, while the technical and subject area staff doing most of the work may not be sufficiently qualified. Having an in-house staff member who is as qualified as the vendor's project manager or at least able to evaluate the work being done will be particularly important in such a case. Without knowledgeable in-house staff who can monitor the vendor's progress, you will be to some extent at the vendor's mercy and may end up with a deficient system. You need an evaluator who understands the technology and can keep the project on track. The in-house project manager should hold frequent meetings with the vendor to obtain updates, assess progress, and address problems as they arise. Keeping minutes at these meetings is also good practice, as this helps keep everyone on track and on the same page.
Everyone wants to get along, and many states have excellent relationships with their vendors. But if a project seems to be getting off track or falling behind, be firm and hold your vendor accountable for failing to deliver. Trust and deference should be balanced against vendor management and the need to get things done on time and to your specifications. Some vendors tend to give clients a bare bones version of their specified product—meeting the requirements of your RFP but not going the extra mile to create the ideal result. Even if the vendor is going to give you a no-frills product, you still want a quality version of what you asked for and were promised. If necessary, do not be afraid to fire your vendor and bring someone else in if the project is not succeeding and the working relationship cannot be salvaged.
Knowledge transfer is the passage of information or expertise from one organization to another, or from one part of an organization to another. From the early stages of your vendor search, your agency should strongly consider knowledge transfer and training. Indeed, a knowledge transfer plan is vital and should be drawn out in advance, perhaps even specified in your RFP, and implemented along the way or once the project is nearing completion. The first step is, make sure you have enough staff in place to maintain what is built. Diligent documentation of everything done by the vendor, as well as by in-house staff, for the entire project is also crucial to efficient knowledge transfer. Additionally, your staff should work alongside the vendor for some time, or receive sufficient training from the developer to understand how the system works. Before the vendor hands off the project, in-house staff should be involved at least once in various processes to learn their structures and contents so that they have a grasp of what the LDS contains, how it is structured and formatted, how to load data, how to make modifications to reflect changing requirements, and other key issues. Additionally, the in-house responsibility must be clearly defined so the transition from vendor to purchaser runs smoothly.
Some states indicate that, once the vendor's main tasks are completed, long-term contracts are not always cost effective, but this depends on your situation. On the one hand, developing in-house expertise may be advantageous so you are not dependent on outside support. On the other, developing the expertise required to fully support a system may be cost prohibitive for your agency, and relying on a vendor may be cheaper in the long run.