Every Process is an Iterative Waterfall

Dave Nicolette raises concerns about organisations moving to agile development while appointing "traditionally-minded" managers. Risk #1 is that these managers turn the agile processes upside down by transforming them into a Staggered Iterative Waterfall as described by Kane Mar.
I deeply agree with the key statement of those articles : agility's main duty is improvement of interactions between individuals with the help of suitable processes. It is not about focusing on a well-polished process aimed at eliminating every possible waste in human activities. However, I do not subcribe to the RUP-bashing movement that seems to result from it.
It does not take a long time to realize that performing sequentially across consecutive iteration the analysis, design/coding and testing activities is a waste of opportunities. In the Staggered Iterative Waterfall pictured by Kane Mar, feedback that could happen inside an iteration takes three of them to arise.
Nevertheless, performing all the activities in the same iteration is not an easy job. For example, in an eXtreme Programming case :

  • The iteration plan needs the user stories to be written and estimated before the iteration start (we have to know which stories fit into the iteration). As a consequence, the time spent by the customer to workout the story must be shortened to its minimum. Every minute spent writing a story without interacting with the developers is a loss of feedback.

  • The customer tests must be run after (and also before) the iteration deliverable is built but before the iteration is over. This implies that customer tests are heavily automated and written as soon as coding starts. Failing to do so means that developers will not be able to produce value while the deliverable candidate is tested.

It is not an easy job but great teams can certainly achieve it to a certain extent.

The question is : can they fully achieve it ?
Obviously not.

In most projects, the best on-site customer (or group of on-site customers) cannot stay on the playground full time and simultaneously being fully aware the underlying customer activities. These on-site customers must be as close as possible to the development team but they have to stay in touch with their own businesses.
The problem is that you cannot put the whole world into a single room.
This is the reason why the stories describing the customer business need to be partly written off-site in an high level analysis activity. This is the reason why the the customer tests cannot be fully written while designing the system. The deliverable must be checked against some off-site exploratory testing activities involving other users.

The combination of those two activities with the software system development enhances the ability of the customer to think about her business process in a true iterative waterfall.
Maybe we can call that "Iterative Business Process Modeling". I'm far from being a BPM expert (just started reading stuff on that subject) but, intuitively, it looks like a good name.
The idea is that two processes, ran on each side, can reinforce each other. Feedback from the real user activity helps the developers to build the expected system while feedback from a delivered system helps real users to think about the way they work.

When going through the iterations, this process can be pictured as following.
The core system development activities are performed in each iteration with agile interactions between individuals at their peak.

Agile Iterative Waterfall

Now let's have a look at the RUP-bashing thing.
The four phases of a RUP iteration look like a miniature waterfall: Inception, Elaboration, Construction, and Transition. [...] The reason RUP is used for iterative waterfall processes and not for agile processes is simply that it is unnecessary in a true agile process; its four phases become unproductive administrative overhead.

So the point is that RUP does not work for agile processes. I could just argue that "If you cannot run a RUP framework in an agile mode then you're not running a RUP framework" but I do not like this kind of extremism.
It looks like there's a big misundertanding regarding the RUP phases. Inception, Elaboration, Construction and Transition are not performed inside a single iteration. They are the phases of a release and each of these phases contains one iteration or more. Inside an iteration, you can do strict Waterfall, you can perform a V-shape or you can blend all the activities in the agile way. You could overlap the iterations but this is considered as a RUP Anti-pattern. It is advisable to do frequent reviews and have multi-skilled teams. Both of them are agile practices.
The RUP phases are the "macroscopic tempo". In a simplistic view of an agile instanciation of RUP, the inception phase is merely the first draft of user stories before development begins. The elaboration phase shows an important milestone : when it is over, the team considers that every "major" user story is implemented. The vision is established and the risk of a project failure is now lowered. The construction phase is about polishing the system until every customer need is implemented. The transition phase hands over the application to the users.

RUP is not inherently less agile than methodologies known for their agility. It is just a matter of usage. Its versatile nature support iterative waterfall processes including a bit of administrative tasks that inevitably happen in the real life with its natural time and space boudaries. Hence, one could even argue that RUP is more able to cope with human concerns than strict methodologies.
Dave Nicolette
>However, I do not subcribe to the RUP-bashing movement that seems to result from it.

I am happy to know you feel this way. However, you may have misinterpreted some portions of my article.

The article includes an excerpt from a piece by Sinan Si Alhir, one of the foremost advocates of RUP, that clearly explains how RUP can be used in an agile way. His diagram illustrates how the four RUP phases can be executed in parallel during the same iteration, in contrast to your statement that they are "phases of a release, and each of these phases contains one iteration or more." I cannot say that I have seen a RUP project function in that way. By the same token, I cannot deny that your organization uses RUP in that way. It is in the nature of adaptive methods that people adapt them.

To be clear, it is in the context of Alhir's model of agile development with RUP that I say the need for four distinct phases simply evaporates, since team members in all disciplines are already collaborating directly. Given that level of collaboration, there is no communication gap that must be address by formal transitions between phases.

You may be relieved to know that I am a strong advocate of RUP in situations where it fits. I consider it the most mature and best proven methodology for lean development. Although RUP can, in theory, be used for "pure" agile development as well, in practice it is not used that way for the reason I mentioned; there is simply no need for the formality of four phases.

If you are running an entire iteration for inception, followed by one or more iterations for elaboration, etc., then you are following a waterfall process but using RUP terminology. I believe that if you are doing this for good reason and you understand what you are doing, then there is no problem. I suspect we are slipping into the trap of arguing about the semantics of words instead of focusing on the meaning behind the words.

Just as RUP works very well when it is applied to the right problems, agile development (especially XP) works very well when it is applied to problems that call for its unique strengths. In those situations, a team can achieve the goals you list in a single iteration. You correctly say it is not an easy job, but I think you would be very pleased at the results achieved by a mature agile team.
"I cannot say that I have seen a RUP project function in that way."
Maybe they were not RUP projects ?
Just have a look, for instance at http://www.ddj.com/dept/arc... by Scott Ambler or http://www.objectmentor.com... by Grady Booch and Robert C. Martin.
"Their" RUP clearly split a release into phases and phases into iterations.

That being said, I totally agree with your conclusion that I had, indeed, misunderstood : it does not make sense to run an agile iteration with four distinct phases. All the activities performed in an iteration must be run in parallel. The serial nature of a development process, materialized by the RUP phases, only makes sense at the macroscopic level : from a set of blurred expectations to a complete product.
Oaz 23 mai 2006 - 23:30
Dave Nicolette
Thank you for recommending those references. I see your point now about multiple iterations within a single RUP phase, and it makes sense.

I think that we, as an industry, are well served by having a range of alternatives for running projects and for managing long-term product development efforts. I am not a believer in a "one size fits all" approach to software development.

I like the final sentence in your recent comment. Very well said!
I share your views regarding the "one size fits all". I'm usually a proponent of agile solutions but I sometimes play the "devil's advocate" when I think the RUP or even a waterfall is more appropriate.
Oaz 24 mai 2006 - 17:13

Fil des commentaires de ce billet

Ajouter un commentaire

Le code HTML est affiché comme du texte et les adresses web sont automatiquement transformées.