From Waterfall to Agile
All too often in life, people form a strong view on a subject, heavily invest energy and emotion, take up cudgels and fight to the death, sometimes literally, to defend their position or impose their opinion. Interestingly, we often find evangelistic fervour when it comes to supporting Waterfall or Agile software development methodologies and their associated project management techniques. For some, Waterfall is old-fashioned, inflexible or unrealistic. For others, Agile practices are a passing fad encouraging ill-disciplined cowboys to develop software in an uncontrolled and unaccountable environment.
Winston R Royce, a Lockheed aeronautical engineer, first described the Waterfall Model in a paper in 1970. His paper, ‘Managing the Development of Large Software Systems’, described a stepwise approach starting with system requirements and moving through the phases of software requirements, analysis, program design, coding, testing and operations. These phases have been reordered, renamed and refined over the years, although the basic concept remains the same. While believing in the concept, Royce recognised that the approach was risky and invited failure. He advocated techniques to reduce risk, including iterations (a feature of Agile!) between adjoining phases, high-level upfront design before detailed requirements were complete and very detailed and enforced documentation along all steps in the process.
Through misinterpretation this model, which came to be known as Waterfall for its ‘step’ illustrations in the paper, was taken up the US Department of Defense in its standard STD 2167 and later the Europeans. Before long, it became the de facto software development standard. Project management frameworks, such as PMBoK and PRINCE2, usually implemented in a process-prescriptive and document-heavy way and implicitly or explicitly acknowledging a phased approach, closely support Waterfall.
Agile software development has a surprisingly long history too, dating back as far as 1957. A number of different ‘agile’ practices, initially inspired from a variety of sources—including notably, lean manufacturing at Toyota in the 1990s—coalesced into the Agile Manifesto in 2001 when a group of software developers came together to discuss lightweight development methods such as Scrum, Extreme Programming and Feature Driven Development, which had also evolved throughout the 1990s.
The Agile Manifesto describes four themes and 12 underlying principles for software development. Working software, adaptability to change, personal interactions and collaboration are more highly valued than prescriptive processes, documentation, contracts, tools or plans. At the foundation of the Agile concept is incremental delivery of the highest business value first, or reduction of the highest risk as quickly as possible through iterative development of software.
In 2004, the Declaration of Interdependence supported the Agile Manifesto due to growing recognition that there was also a need for a broader statement addressing project management and general leadership. The statement, empathetic to the principles and practices behind Agile thinking, focused on value, customers, individuals, teams, uncertainty and context.
Benefits and drawbacks
So, what are the benefits and shortcomings of each approach? Well, Waterfall is easy to understand and communicate to a range of stakeholders. There is natural appeal in its simple, linear, structured approach, which usually combines formal phase gating by key stakeholders, preventing the project progressing until satisfied the previous phase has been properly completed. Greater time spent in requirements and design is seen to reduce later coding and testing costs. Waterfall planning is predictive and can therefore inform the development of longer-term business plans and road maps, although longer timeframes also engender greater uncertainty. Waterfall appears more disciplined than Agile, but Agile enforces more discipline than is immediately obvious.
On the downside, Waterfall is less adaptable to changing business requirements, which occur when at the outset business people do not know exactly what they need, or a changing market compels a change of direction. There is also greater emphasis and effort devoted to following prescriptive process or transient project control documentation, which contributes little or no value to the required business goals.
Agile acknowledges requirements’ uncertainty or rapidly changing market conditions by working in short iterations to deliver incremental business value. This is facilitated by small, cross-functional teams that combine the complementary skills and experience of business and IT people. These teams are self-disciplined and self-organised although they take their business priority direction from the key business stakeholder. Project managers often take on the role of servant-leaders, facilitating information exchange, protecting the team from interference and removing progress blockers rather than directing the team.
Agile has its weaknesses too. There is a heavy reliance on close business involvement, ideally on a daily basis. This places a great burden on business people who must also juggle daily operational responsibilities. Teams are more efficient if they are small, co-located and have the right physical working environment, which is often hard to achieve in large, geographically separated organisations. It can be a lot harder to scale up and control Agile teams for larger software development efforts or to integrate them with other areas of a business that do not operate using these techniques. Agile is not very effective in integration projects or particularly valuable where requirements are well understood and stable.
It is important to be agile and not just do Agile. Company culture, individual personalities, the nature of the project and a range of other factors should be taken into account when choosing a development and management method. In recent years, work has been done to combine the best of predictive and phased approaches with adaptive and iterative approaches forming hybrid methodologies such as the Rational Unified Process or the lightweight Agile Unified Process. Applying these or adapting in-house development methodologies and project management practices to the business circumstances are the third way. The answer is not binary!
