Not so Agile: Part II – How to transform the software development process
In the first part of this 2-episode series on software development, we discussed the challenges posed by both Waterfall and Agile (scrum) methodologies and why using these could lead to a loss in productivity.
We had promised to come back with a new solution that can overcome these challenges and yield very high productivity jumps. So, in this episode, we have invited Anantha Keerthi back to describe the new "Rapid Feature Flow Model" innovated by Vector. He discusses why adopting the model's 'flow principles' can help companies increase productivity by 70-90%, reduce rework by 50-80% and increase release frequency by 2-3 times. All this while creating a stress-free work environment for developers!
For a complete understanding of the core issue that afflicts many of today’s software development environments and the solution visit https://www.vectorconsulting.
Hello and a very warm welcome to Counterpoint Podcast. I am Shubham Agarwal. This podcast is a continuation of our previous episode #42 on challenges in software product development. We discussed about Sprint based Agile approach, some of the flawed assumptions behind current methodology and its devastating effect on the overall software project. The root cause for long & unpredictable release time, huge backlog of bugs, budget overrun or high stress close to product release. In case you haven’t had a chance to listen to the episode, I would strongly urge you to do so to understand the problem statement in depth.
In the previous session with Anantha Keerthi we focused more about the problem statement – In today’s episode let us explore the solution part. Anantha Keerthi is a partner with Vector Consulting Group and has transformed many systems in the past. So, let’s welcome Anantha,
| Shubham Agarwal : | Hi Anantha, welcome to the counterpoint podcast. |
| Anantha Keerthi : | Thank you Shubam, Thank you for having me. |
| Shubham Agarwal : | Anantha I can’t forget the interesting analogy you gave about Procrustes to explain the vicious loop created by Sprint cycles and also the constraints of SMEs (subject matter experts in Software teams), can I request you to quickly tell us the vicious loop for a quick revision and tell us how do we come out of this vicious loop. |
| Anantha Keerthi : | One of the key differentiators proposed by Agile was iterative software development, Iterative development by definition is the opposite of pre-planning. Isn’t it? ….. so then, how can you plan an overall project or version delivery? Any iteration can go endless! ……. So, to address this problem, there is a solution in Agile framework. i.e Sprint deadlines. Like Procrustes’s standard iron bed, the sprint cycle forces people to somehow close things.
Every sprint becomes a deadline with pre- committed scope. As almost all resources in the team become extremely busy towards sprint closure, any planning activity of subsequent sprint is left behind. Bcos There is no time! ….Consequently, in every sprint, initial time is used up in planning work. Actual development and testing happens towards second half of a sprint cycle, this builds pressure of time towards end of every sprint. So what happens when there is pressure of time? ….it leads to skips and misses (compromise in Requirement & Design), yes !. any such compromises will come backs as bugs, regressions & rework, now the team becomes even more busy (especially the subject matter experts). So, there is almost no time for preparation for subsequent sprint. Now every sprint starts without adequate preparation. This vicious loop goes on. One of the assumptions of the waterfall method, which was questioned by Agile, is the ability of customers to clearly specify requirements upfront! Many times, a requirement becomes clear only when customers actually see or start using some piece of your product!!. In this situation the need to have a perfect spec from the customer can make the requirement phase completely out of control. But at same time, the agile implementations also suffer from inadequate requirements and design. Yes, we had a detailed discussion about this in the previous episode. If you carefully see, we have a generic conflict underlying the entire software development process. Can you guess what is it? |
| Shubham Agarwal : | Let me Try. Should we freeze the requirement before starting the coding or not. I am I right ? ● On one hand, we should freeze requirements & design before implementation (ie. coding) because this approach ensures rework is under control and hence, time is under control. ● On the other hand, we should not freeze requirements & Design because it is impossible to freeze them. |
| Shubham Agarwal : | Users get clarity on what they actually want after using or visually observing the product. |
| Shubham Agarwal : | Then how do we come out of this conflict? |
| Ananatha Keerthi : | This conflict has given us an important insight about two kinds of rework …..or errors which gets generated in any new product development process.One you can call it as Type A (or Foresight rework) and the second is called as Type B rework (or hindsight error) Type A rework is created because of skips and misses made under pressure of time or lack of synchronization between the team members. Whenever you see these kinds of errors you will say – “Oh! how can this happen, we could have prevented this”. That means the process or knowledge to prevent this error already exists within the team, nevertheless the team is committing these errors under pressure of time. Shubham Agarwal – So in a way, type A is a Preventable error. |
| Anantha Keerthi : | Yes! so Using testing team to give feedback on these types of preventable errors is a criminal wastage of capacity isn’t it? So, type A errors always causes frustration among the team or the management. Shubham Agarwal – So what is the second type of rework or errors |
| Anantha Keerthi : | The second type of rework i.e., Type B: these errors are detected only after the event of testing is done, only in the hindsight you can tell what the real cause for these kinds of errors was. That is why type B errors can also be called as Hindsight errors. No amount of thought experiment can help one visualize the Type B errors upfront. The other way to look at is, type B errors create new knowledge which was not there before. The earlier you detect, the better it is. (In fact Proto types are created for the same reason.) |
| Shubham Agarwal : | In a nutshell – Type A is an avoidable rework, this got generated because the team has violated some internal process under pressure of time, Whereas Type B error can be detected only after final testing. |
| Anantha Keerthi : | That’s right ! So, Waterfall method somewhere implicitly assumes that all errors are Type A (Preventable errors), hence it insists on adherence to process, For example – “If you want me to start the development, I will say… first sign off on the requirement, sign off on the spec, sign off on the design !! this way in waterfall approach there is lot of checks and gates before moving from one stage to another. …… Whereas Agile, somewhere implicitly assumes that all errors are of Type B, hence it considers one-time handovers between phases as a waste of time. therefore, agile propagates iterative development – The school of thought here is “I know requirement cannot frozen…. Let’s get started, with more iteration it will mature” hence in agile it’s alright to have frequent back and forth between all phases. |
| Shubham Agarwal : | Oh! This sounds really interesting – you are now clearly verbalizing the inherent assumptions of both Waterfall and Agile approach, its getting very clear and simple to understand. |
| Anantha Keerthi : | If you go back to History of Automobile manufacturing this “Agile” way is exactly how automobiles used to be manufactured in late 18th century. Have you heard of Craft manufacturing ? |
| Shubham Agarwal : | Yeah. |
| Anantha Keerthi : | A car used to get manufactured from start to end through a group of expert resources with different skills. This cross functional team is expected to cohesively work and eventually a vehicle would come out after spending lot of time in iterative work. Ofcourse ! This way of car manufacturing was very expensive and only the rich could afford it …..in that era. |
| Shubham Agarwal : | Then came Henry Ford |
| Anantha Keerthi : | When Henry Ford presented the assembly line concept with clear division of labor, The entire car manufacturing was transformed. There was a Clear sequential working with different resources specializing in different types of assembly task. The cars moved between stations (instead of the workers moving). The specialization on one type of task,… and by avoiding switching losses between different type of work… enabled a productivity jump by multiple times.
The revolutionary idea dramatically reduced the cost of making a car. The masses could afford a car because of the enhanced productivity. In a way, the car industry gave up the “agile methodology” of car making and achieved giant strides in productivity. |
| Shubham Agarwal : | – So, Are you advocating assembly line working for software development? |
| Anantha Keerthi : | – Well, before we going there first we need to understand, under what boundary conditions the assembly line works..
Look! an assembly line approach requires two critical conditions to be successful 1. Flow backs are negligible. Work moves forward in one direction. If nature of work is such that it frequently comes back to previous workstations for rework (it is called as flowback), such flowbacks can create chaos with uncontrolled line stoppages and reduced output. (In such an environment, craft manufacturing, or the agile method, is a better way). |
| Shubham Agarwal : | Are you saying, If we meet these two conditions i.e., 1) very minimal flow backs and 2) small transfer batches then we have a solution ? |
| Anantha Keerthi : | – with this we can resolve the inherent conflict of the software development process.
What is the desired work-flow in a software development – First you need to understand the requirement, elaborate the spec, then do a proper design, then only get in to development and finally testing. As discussed before, the major reason for flowbacks is improper Requirement detailing and Design. So, Reorganizing the team into Recruitment assessment team, Design team, development and testing teams – This helps to create the required division of labor for assembly line working. Now we have clear work centers as Requirement and design in upstream and development and testing in downstream. The question is? Who should be working in upstream and who should be working in downstream ? |
| Shubham Agarwal : | I think it is SMEs in upstream |
| Anantha Keerthi : | – Obviously ! the experienced SMEs are the best resources to work in upstream. Such expert resources are limited in every team. when your expert resources are dedicatedly work in Requirement assessment and design, the need for them to go down stream is very less. isn’t it ? – i.e |
| Shubham Agarwal : | – they don’t have to waste their capacity in development (coding, bug fixing etc). |
| Anantha Keerthi : | So, they are less busy now. |
| Shubham Agarwal : | What else is expected out of the Requirement team ? |
| Anantha Keerthi : | – This is not just about understanding the requirement from customer perspective or from development perspective, …This phase should focus not only defining what should be done in a feature but also defining what should not be done, or what will not be given. This could be the first workstation but it is also important to begin with the end in mind. after understanding the requirement also define how this will be validated in testing. |
| Shubham Agarwal : | Inputs from testing team is required. |
| Anantha Keerthi : | Yes, it is important. At times, visual prototypes can also help customers visualize what they really want. |
| Shubham Agarwal : | How about Design team? |
| Anantha Keerthi : | Surfacing the various dependencies, interaction with other functionalities, Impact analysis, Architecture, infrastructure Review, defining the Boundary conditions – all this constitutes design work which is aimed at preventing Type A errors. To a large extent we can pre-pone Type B errors as well. Regressions are minimized as dependencies are clearly defined. |
| Shubham Agarwal : | Earlier you mentioned that, when it comes to engineering of physical products, these design aspects are never ignored. |
| Anantha Keerthi : | Yes ! in physical products, the damage is visible, damage in terms material, dismantling of structure, rework, everything is visible. But most people think, software product development is more forgiving !! The reality is – The damage is just hidden. |
| Shubham Agarwal : | This is really great, we are transforming a project environment into a flow environment. |
| Anantha Keerthi : | So, far we have discussed about, minimizing the flowbacks. There is one more aspect of Rapid feature flow model, i.e., Minimizing the transfer batch In a high variability environment, if we impose rigid time buckets, it’s a clear recipe for skips and misses. Software development is a high variability environment, if we impose sprint time buckets it creates skips and misses |
| Shubham Agarwal : | it leads to Type A errors. |
| Anantha Keerthi : | That’s right. The best way is to replace the sprints with work in progress control rule i.e WIP rule. This needs to be applied across each resource group – Like you have Requirement group, Design, development & testing groups. A new feature will be introduced to a resource group only when one under progress is complete. The rule of “One-Out-One-In” ensures a single piece transfer batch. This is how the second important criteria of high productivity assembly line is met – that is minimizing transfer batch. |
| Shubham Agarwal : | Oh ! this is so different ! Suppose I am in Development team, I can pickup a new feature for working only when I finish which i already have !! |
| Anantha Keerthi : | Yes ! Here, completion decides the start ! the entire focus is on completion!. |
| Shubham Agarwal : | Its means there must be a clear definition of what is complete for each stage. |
| Anantha Keerthi : | – Yes! It is called as closure criteria… Not only that,…. since you are not allowed to pickup new feature until you complete something in-hand, there is a waiting queue of features or work packets building-up outside the WIP bucket. When you finish one, …which one are you going pickup from the Queue. |
| Shubham Agarwal : | Correct ! I never thought about it ? |
| Anantha Keerthi : | There must be a clear sequential priority for each feature or workpackets Following the above principles…. will help in rapid flow of features from this system. That is why we call it as Rapid feature flow Model! |
| Shubham Agarwal : | Anantha, what kind of results can we expect from the new rapid feature flow model. |
| Anantha Keerthi : | A typical implementation must see a drastic reduction in rework (number of bugs generated, number of regressions, reopen cases), you can expect a reduction of 50% to 80%A productivity jump of 70 to 120% is not surprising – because this is the effect of high productivity assembly line; Increase in output corresponds to reduction in lead time as well, that means frequency of software release also drastically improves by 2 to 3 times. |
| Shubham Agarwal : | What kind of timeline is required? |
| Anantha Keerthi : | Around to 2 to 3 release cycles are required for the new process to mature and stabilize. |
| Shubham Agarwal : | Are there any drawbacks or limitations of Rapid feature flow model? |
| Anantha Keerthi : | The WIP rule is an anti-scheduling mechanism. While following the WIP rule, a new work packet is taken up only based on completing something in hand., not because time has come. That means you can’t follow task scheduling. Then how do you predict the release date? Implementation of flow model exposes queue at each work center, also it is easy to determine clearly the flow rate at each work center; using this combination we can compute the Expected time of arrival (ETA) not only for each feature but also for the entire release. Like a google map. |
| Shubham Agarwal : | Great! that means this model allows much better predictability without scheduling ! |
| Anantha Keerthi : | Not only this. Since this a perpetual feature flow system, Single features or a set of features can be bundled into releases as and when you want. Instead of committing to a release plan upfront for a year, marketers can decide on release strategy based on dynamics of the market.This provides a lot of flexibility to react to competition strategy, you can change the queue of features in the waiting list to deal with market dynamics. |
| Shubham Agarwal : | You have much better predictability and also flexibility. |
| Shubham Agarwal : | Fantastic. Anantha it was a very insightful interaction, Thank you for introducing us to this new Rapid feature flow model. It is truly innovative. It combines the inherent advantages of Waterfall and Agile Models, while overcoming the disadvantages. |
| Anantha Keerthi : | Yes !This model 1. Allow for flexibility and speed as per Agile 2. Prevent Type A errors & rework as per Waterfall 3. Also recognize the constraint of the system (as per TOC). |
| Anantha Keerthi : | – Yes ! |
| Shubham Agarwal : | Thank you once again. |
| Anantha Keerthi : | Thank you, Shubham. Thank you! |
Related Information
Get in touch
Vector Management Consulting Pvt. Ltd.
10th floor, Thane One, DIL Complex,
Ghodbunder Road, Majiwada,
Thane (West), Maharashtra - 400610, India.