Dispatching Failure: The LASCAD Project

Dispatching Failure: The LASCAD Project

CST270 - Systems Analysis & Design
Robert Pinchbeck
Term Paper
May 10, 2006

In 1990, The London Ambulance Service began planning to implement a Computer Aided Dispatch system in response to overwhelming public pressure to improve their emergency response times. Their goal was to replace a wholly manual process with an automated system that would bring their response times into compliance with national standards. Yet despite the use of formal processes and development methodologies, the project was a failure. The computerized system did not work as expected and the London Ambulance Service was forced to return to their original manual process. A formal inquiry into the project revealed many factors that contributed to the project failure. Ultimately, however, the essential cause of the failure was a single management decision.

At the time, the London Ambulance Service (LAS) was the largest ambulance service in the world -- covering an area of more than 600 square miles with a resident population of almost 7 million people. On any given day, LAS transported more than 5,000 patients and handled more than 2,000 telephone calls. Employing over 2,700 people, including an operational staff of roughly 2,000 people, LAS was capable of deploying a fleet of over 750 vehicles. Given its sheer size, LAS faced considerable challenges in identifying available resources and deploying them in a timely manner. To this end, they employed a manual system using paper forms to track incidents and monitor resource availability. It was believed that a fully automated solution would ease the burden of this manual process and allow the entire organization to function more efficiently.

To understand the requirements of the automated system, it will help to consider the manual system that was in place when the computerized system was designed. Initially, telephone operators would answer phone calls, write down the details on a paper incident form, and consult a map book to find the incident location. The paper form would then be physically passed (via conveyor belt) to central reviewers who would identify and remove duplicate incidents. The central reviewer would then pass the incident form to the appropriate regional resource allocator. The resource allocator would search through paper availability records to find an appropriate vehicle and crew to assign to the incident. Finally, the form would be passed to a dispatcher who would contact the appropriate people (often by telephone) and deploy the assigned vehicles and crews. National standards set by the Operational Research Consultancy (ORCON) state that this entire process should take no more than 3 minutes. The manual paper process was taking significantly longer than this nationally accepted standard.

Computer Aided Dispatch (CAD) systems are meant to ease the process of deploying a fleet of vehicles and crews in response to incidents. As such, CAD systems usually include features that assist with taking telephone calls, recording details, locating incidents, identifying available and appropriate resources, deploying those resources, and communicating incident details to response teams. CAD systems are also meant to help supervisors anticipate resource demands and assess the performance of response teams. The London Ambulance Service Computer Aided Dispatch (LASCAD) system was designed to accomplish all of these goals. Moreover, it was designed to automatically identify, and propose deployment of, available resources in response to incidents with little human intervention or involvement. This latter portion of the LASCAD design had never been attempted by any emergency service in the United Kingdom, and posed considerable risk to the success of the project.

Interestingly, this was not the first time that the London Ambulance Service had attempted to implement a CAD system. A similar project had been abandoned shortly before work began on the new system. In telltale fashion, the earlier project was abandoned after it was determined that it could not handle the daily demands of LAS. Consequently, a new team was assembled to design and build a better system. The planning team consisted of the Director of Support Services, the Systems Manager, a contract analyst, and the Control Room Services Manager. Many areas of LAS were included in the planning process, yet the planning team excluded the people who would be most affected by the design of the system -- the ambulance crews. This was a significant mistake that further diminished the cultural feasibility of the project.

Initially, the planning team investigated the possibility of tailoring existing CAD systems to LAS needs. They considered three systems that were in use by other regional emergency response services, but found the prospect of converting those systems to be cost prohibitive. Given the breadth of the LAS coverage area, it is understandable that the solutions employed by smaller ambulance services would be unsuitable. Still, it is hard to understand why the London Ambulance Service didn't investigate tailoring solutions that were being used by ambulance services in other large cities around the world. After a short investigation, the planning team concluded that a new system would need to be built, and work began in earnest on a detailed System Requirements Specification (SRS). The system, as specified, would include computer aided dispatch, computer map display, and automatic vehicle location. The completed system would require integration with several other systems, including a computerized radio communication system, an automatic vehicle location system, and a mobile data terminal system. The requirements specification called for integration of all of these systems with the CAD software, but did not specify, with appropriate detail, how this would be accomplished. It is important to note that the specifications for the new system also resulted in a new Operational Method of Working that would significantly alter the daily work duties of call-center staff and ambulance crews. Despite the fact that this would impact their duties, ambulance crews were again marginalized during the process of developing this document.

When the System Requirements Specification was completed in February 1991, the project team began the process of contracting suppliers. They advertised the project in The Journal of the European Communities and 35 potential suppliers responded. The respondents were sent a copy of the SRS and given an aggressive and non-negotiable deadline of January 8, 1993 -- less than 11 months. After considering these documents, 17 suppliers submitted proposals and, of these, only 5 claimed that they could meet the deadline. The lowest bid for the entire project came from a consortium formed by Apricot Computers, Datatrak, and Systems Options (SO). SO would develop the CAD software that would ultimately interface with the other integrated systems (implemented by Apricot Computers and Datatrak). At Ł937,463, their bid was nearly 44% lower than the next-lowest bid. It is important to note that Systems Options was a small software consulting company with no prior experience with Computer Aided Dispatch systems development. This may explain why their bid was so much lower than their nearest competitor (McDonnell Douglass). Yet despite this substantial difference in price, and a substantial difference in the capabilities of the software suppliers, LAS awarded the contract to the lowest bidder without any real investigation into the qualitative differences between the lowest competing bids.

The Regional Health Authority Standing Financial Instructions state that all procurements should be advertised publicly and that supplier contracts should be awarded to the lowest bidder unless there are "good and sufficient reasons to the contrary." This is certainly an appropriate approach for fungible commodities when quality will not vary much between suppliers. However, in the case of large Information Technology purchases, the quality of the resulting system depends entirely on the abilities of the supplier to implement the system properly. Given the scope of the requirements, it would have been wiser to award the contract to a larger, more experienced, albeit more expensive, supplier. This was the crucial mistake that would inevitably doom the project to failure. Had the project team retained a more capable, properly staffed, and better quality supplier, the project might have had a chance of success. Unfortunately, LAS favored price over quality and got what they proverbially paid for. It is hard to understand why the project team made this choice, given that the previous attempt at CAD had resulted in failure, and that the previous failure was due to the "inability of the software supplier to understand the requirements of the system." Their decision to award the contract to a comparatively less experienced consulting firm is especially surprising given that the new project would be far more complicated than the previous attempt.

Having been awarded the contract, Systems Options (SO) began working on a comprehensive System Design Specification. This design specification took roughly two months to complete and was delivered in July 1991, leaving less than 6 months to implement the CAD software. Project management was provided almost entirely by LAS, who selected the PRINCE (Project in Controlled Environment) Project Management Methodology for the project. Incredibly, the project management team had no experience with using this methodology, and little real experience with project management in general. Training in the PRINCE methodology was provided, but the team was unable to use the methodology effectively. Moreover, the project management team was not even assigned full time to the task of project management. In fact, LAS did not assign any of their employees to the project on a full-time basis. The lack of expertise, combined with distractions from other duties, led to a general lack of project management that contributed to the failure of the project. Had the project team retained a full time, more experienced project manager, they would have been able to identify major project problems earlier. Better project management would not likely have allowed the project to succeed, but it would certainly have made the imminent failure apparent much sooner.

With less than 6 months to finish the project, Systems Options began to miss deadlines. They routinely delivered software late, and what they did deliver was riddled with defects. Worse still, SO routinely made changes to the software (at the behest of users) without any formal revision and review process. This led to further problems, when tested portions of the system were "secretly" modified without being retested for defects. Remarkably, despite its mission critical nature, the software project did not have a full time Quality Assurance (QA) team. Systems Options had only one part time employee performing QA for the duration of the project.

In order for Quality Assurance to be effective, it must be independent of the development organization and empowered to change deadlines when software does not meet quality standards. This is vitally important for mission critical software, especially when human lives are at stake. The Quality Assurance provided by SO was inadequate and, by its very design, presented a conflict of interest. Systems Options was under significant pressure to deliver quality software on time and, therefore, had a no incentive to find and report flaws in their own work.

By December 1991, it was clear to the project team that the software would not be ready for full deployment on January 8, 1992. Indeed, full system testing had not commenced prior to the deadline. In January 1992, Systems Options made their first attempt at system testing. However, the tests that were performed were not truly valid because the full system was not yet implemented; there were gaps in all of the other integrated systems. Furthermore, the tests that were done did not incorporate the possibility for integrated system errors and failures. This is very surprising, since these types of problems were already present prior to system testing. Testing a system without incorporating the possibility for errors and failures will only indicate how the system will perform in a 100% reliable environment. It is relatively simple, and certainly appropriate, to incorporate occasional (or even frequent) simulated failures into system tests, yet Systems Options did not do this. Perhaps this is because, if they had, LAS would have discovered that the system was not as fault tolerant as it needed to be. Had the project team retained an independent Quality Assurance team to test and verify the system, they would have discovered this fatal flaw much sooner.

When the project team officially accepted the fact that they were not going to meet the project deadline, they decided to deploy the LASCAD system in phases, beginning with the only portion of the system that was really working. Phase 1 would allow telephone operators to use the software to take calls, enter incident details into their computers, and use a computerized gazetteer to locate incidents. Paper incident forms would then be printed out and dispatched via the original manual paper process. Phase 2 would allow for electronic delivery of incident details to resource allocators. For the first two phases, human decisions would still determine which vehicles and ambulance crews were dispatched in response to any given incident. Phase 3 would eliminate most human decisions and allow telephone operators to automatically deploy vehicles and crews in response to most incidents.

Except for a single incident form which was lost when a printer was accidentally turned off, Phase 1 was basically a success. The call-taking, data entry, and gazetteer portions of the system worked effectively. Unfortunately, in anticipation of a single phase changeover, LAS had reorganized the call center's physical layout and removed the conveyor belt system. This is truly demonstrative of the unwarranted optimism, and lack of backup planning, that had plagued the project from its inception. As a result, it became necessary to employ "runners" to physically move the paper forms from the computer printers to the central resource allocators. Phase 2 eventually eliminated the need for moving paper, but still required central resource allocators to consult paper records to locate available resources.

Phase 3 was fully implemented on October 26, 1992. In the months leading up to this phase, the LASCAD system underwent a process of continual change; the system was never stable. Indeed the defect tracking system still had open issues during the deployment of all three phases. It is hard to understand why an emergency response service would jeopardize human lives by deploying a verifiably defective CAD system. Perhaps the best explanation is that the management team was facing persistent pressure to succeed and, as a result, was not willing to ensure quality by extending deadlines.

Phase 3 was the first time that the system operated for the whole of London. It was a total failure. On November 4, after operating for only 9 days, the system came to a complete halt. Backup systems failed to engage when the primary system failed. These backup systems were part of the original design, yet had never been tested. In the end, the system failure was caused by a minor programming error in a section of the system that had also never been tested. Given the publicity and magnitude of the system failure, LAS was finally forced to admit defeat. They retired the LASCAD system and restored the original manual paper processes. It would be another 4 years before they would finally deploy a Computer Aided Dispatch system that worked.

As could be expected with any major project failure, many mistakes were made during planning and implementation of the LASCAD system. In many projects, these mistakes are often individually survivable, but they become fatal to the project in aggregate. This is not the case with LASCAD. There was one simple and crucial mistake that was made, early in the project, which caused its eventual and inevitable failure. The essential mistake was retaining a small and inexperienced software consulting firm to develop a mission critical system. Indeed, there were other mistakes that, in hindsight, made the failure worse. For example, LAS should have included ambulance crew representatives in the planning and design phases of the project. They should have arranged for better training of their staff and ambulance crews. They certainly should have employed full time, experienced, project management and quality assurance teams. However, if LAS had contracted a larger, more qualified, software supplier who possessed the foresight to anticipate and proactively respond to potential problems, and who employed the resources to provide excellent project management and superior quality assurance, then the LASCAD system might have had a chance of success. All other mistakes would likely have been anticipated, prevented, or at least rectified, by a superior software supplier. Even the aggressive timetable might have been met, given a supplier with the experience and the resources to do so. Unfortunately for LAS, their early decision to retain an inappropriate software supplier would make their already aggressive deadline impossible, and ultimately lead to the total failure of the LASCAD system.

The London Ambulance Service was being realistic when they designed the LASCAD system. In none of the prevailing literature is there any faulting of their System Requirements Specification, or the System Design Specification developed by Systems Options. Their deadline was considered aggressive, but possible, by 5 of the suppliers who bid for the project. The project team and the contracted suppliers observed many formal processes and methodologies. The project planners followed the standard systems development life cycle and wrote a detailed requirements specification. They followed standard financial instructions for their organization and they objectively compared each of 17 bids against a set of weighted criteria before contracting a supplier. They selected the PRINCE project management methodology and formally trained their staff to use it. They followed the systems development life cycle again when they had System Options prepare a detailed design specification to ensure that the requirements were understood. There was admittedly limited Quality Assurance, but a formal process existed to report and track defects in the system.

Evidently, formal processes are not enough to ensure success, particularly when those processes encourage poor decisions. Ultimately, it is the practitioners of formal processes and methodologies who will determine the fate of any given undertaking. In order for any formal methodology to succeed, the people who practice it must have the experience, the skill, and the foresight to understand the significance of that methodology and, more importantly, the significance of their own decisions.

References

Finkelstein, A. & Dowell, J. "A Comedy of Errors: the London Ambulance Service case study" in Proc. 8th International Workshop on Software Specification & Design IWSSD-8, (IEEE CS Press), 1996, 2-4.

McGrath, K. (2001). "The Golden Circle: A Case Study Of Organizational Change At The London Ambulance Service." London School of Economics and Sociology, London, 29 June.

Page, D., Williams, P. and Boyd, D. (1993). "Report of the Inquiry into the London Ambulance Service." South West Thames Regional Health Authority, 25 February.

"Why Software Systems Fail." Planet Papers. Retrieved April 13, 2006, from the World Wide Web: http://www.planetpapers.com/Assets/1931.php

"CAD Failure LAS. 1992." Retrieved April 18, 2006, from the World Wide Web: http://www.lond.ambulance.freeuk.com/cad.html