Software companies have used waterfall methodologies for decades now. Agile has made major inroads in the product development area but still its a long way to go for non product development related IT projects. A classic example are software vendors with commercial off the shelf (COTS) products. They have been using implementation methodology with their customers for a long time and it seemed to work until so far. But the cracks are emerging and there is a growing noise to do something different. Change is never easy or without problems so before a company embarks on the agile journey thinking of it as a silver bullet to solve its miseries, a thorough analysis of the current challenges is essential.
Not all your problems can be attributed to the waterfall methodology and these are the ones moving to agile won’t solve. Though there are a few it can and they can have a significant impact on your success rate. Typically four common challenges emerge that software companies face with the waterfall model :
Scope closure and/or creep
Waterfall expects to have the project scope unambiguously agreed and signed off with the customer to begin the design and development phases. Once done, its set in stone and further changes are to be managed as “Change Requests” which essentially mean more cost to the customer. If they are business critical and need to be absorbed in the current project, regardless of who pays for it, its likely to result in a delayed timeline. For large-scale implementations, this essentially means that once the customer signs a particular scope, for months or maybe years, they won’t be able to suggest major scope changes. This is impractical and unrealistic considering today’s dynamic and competitive business world. For implementation projects with COTS as a base, the customers do not have clear understanding of the base product to be able to elicit requirements explicitly and fully at the start of the project. Due to this, they are scared to sign off the scope leading to many prolonged and highly delayed scoping sessions. Scope management is one of the key challenges with waterfall. This was aptly illustrated by a key study of 1027 IT projects carried out in United Kingdom by in 2001 who reported that “scope management related to attempting waterfall practices was the single largest contributing factor for failure.”
Timely customer feedback
With waterfall, the customer is involved at the beginning of the project during scoping and towards the end at the acceptance testing stages. Between these for a prolonged period, the software vendor works in isolation. Customers get a first hand experience of the software only towards the end stages when all development is done and a lot of budget / time is spent. When they see it in action, they start to realize what they are getting and give valuable feedback to improve the system – to make it more user-friendly or efficient for their business. But it’s very late in the game and again the dreaded change request route is suggested. Either the customer would agree to shell out more or accept less than efficient system. In both cases – it’s not a very pleased customer.
Changing business behavior
Large scale software implementations often times need a change in the business processes for the customer, but the system is modeled either on the existing business model or a design of how the new business model should be. In either case, when the system actually hits ground zero in the hands of end users, a lot of gaps are discovered which need to be bridged by the dreaded change requests again.
Doing it right the first time
Linear development model using waterfall leaves no room for error. The scoping and design phases have to be perfect with a clean development followed by perfect acceptance testing, system integration and deployment. Waterfall forces us to get it right first time but for complex system implementations, this is practically impossible. There are challenges faced at every stage leading to planning and re-planning eventually delivering projects a higher cost and timeline than planned. It’s not much of a surprise to anyone actually and so buffers in cost and timeline are built to absorb these – which essentially means a not so satisfactory experience to the customer. They could have it for less cost earlier.
It is no surprise that faced with these challenges, companies look to Agile to be their savior which in principle at least suggests a logical path forward. It changes the paradigm by welcoming scope changes even late in the game, built-in regular feedback mechanisms and close customer collaboration at all stages to name a few.
Waterfall evolved as the structured way of working from the code-use-fix-code again methods at the dawn of software industry. It did serve its purpose and everyone learnt a lot. Not all is to be thrown away, but we must evolve to better ways of developing software. The journey from waterfall to agile is again our journey of evolution – the old must make way for the new, the better.