Episode 42
Not So Agile: Part I – Challenges in Software Development
Category : High Variability Operations
In this podcast we visit the challenges posed by Waterfall and Agile (scrum), the most popular methodologies for software development. While examining Agile, Anantha Keerthi helps us question some of the assumptions behind few of its practices like - working with small ‘self-sufficient’ teams, keeping design and requirements fluid & use of standard sprint cycles. He describes how by understanding the wrong assumptions, and by exposing the generic conflict underlying all software development processes, a truly agile software development method can be evolved.
Read more about the generic conflict underlying all software development processes here: https://www.
Transcript
| Shubham Agarwal : | Hello and welcome to the Counterpoint Podcast. Today we’re discussing about a very popular methodology in software product development. It’s called Agile. Now agile basically places an extremely strong focus on rapid response to the customer and making speed as the competitive edge. It aims for early and continuous delivery of valuable software, embracing change, frequent delivery, and ends with sustainable development, self organising teams and regular reflection and adjustment. Now, while the methodology was targeted towards achieving quick turnaround times, it is not so agile today. Why So? What are the reasons behind it? What are some of the core assumptions which do not hold good anymore, is what we are discussing today with Anantha Keerthi. He’s a senior consultant with the Vector Consulting Group and has transformed many systems in the past. So let’s welcome Anantha Hi, Anantha, welcome to the counterpoint podcast. How are you? |
| Anantha Keerthi : | Thank you. Thank you, Shubham. How about you? |
| Shubham Agarwal : | I’m great. Thank you so much for being here. |
| Anantha Keerthi : | Yeah. Nice. Nice, nice talking to you after a very long time. |
| Shubham Agarwal : | Right, great. So Anantha we’re going to start with the very basic question on agile, I think something which everyone would want to ask, since the introduction that we have given, what is it that is wrong with Agile? |
| Anantha Keerthi : | Yeah, so Shubham, I know this is a very interesting topic. What’s wrong with Agile is what you asked. Have you heard of a one size fits all approach? |
| Shubham Agarwal : | Yes, I have |
| Anantha Keerthi : | Yeah. Suppose you have only one size shoe. And if you apply this rule, whether it is an ant or an elephant, see, sprint based Agile methodology is suffering from something similar to this. So let me give you a much weird analogy. So that you can remember this, okay. So in Greek mythology, there was a character called as Procrustes. Okay, Procrustes was nothing but a robber, his, you can call him as a bandit. He used to torture his victim by force fitting them into a standard iron bed. See, if the victims were shorter than the bed size, he would forcefully stretch them. And if they’re taller, he would cut off their limbs as per the size of the bed. |
| Anantha Keerthi : | That’s why I’ve told you this is a very weird analogy. And you’ll definitely remember this |
| Shubham Agarwal : | Yeah you are right, i will never forget this. |
| Anantha Keerthi : | So in an environment like software, where there is a high variability of scope, high variability in terms of efforts, right variation in terms of degree of complexity, if you use a standard scale of time to force fit all kinds of scope, it’s nothing short of a torture technique, isn’t it? |
| Shubham Agarwal : | Correct? I agree. |
| Anantha Keerthi : | So sprint based agile approach is suffering from Procrustes phenomena, because it uses the scissor of time to cut the scope. See, so, in fact, agile approach was introduced as a solution to certain problem that was faced by the software product development community at that point of time. Okay, so you’re asking me what’s wrong with Agile? Yeah, the right question to ask according to me is what are those problems for which agile approach was considered as a solution? Whether companies are still facing these problems even after implementing agile approach? Right, |
| Shubham Agarwal : | Right, right. No, I love it when we have to go back to the basics and you know, start from the start. So, I will I will rather pose that question that why did the need for agile come in as in what are some of those major issues that were faced by companies in software development? |
| Anantha Keerthi : | See, well, as you know, the the software technology has definitely impacted the productivity across every industry, okay. There is no one industry, what we can say that it is not impacted by software industry. But if you if you observe the productivity of software industry itself, it is pathetic. The project success rate in software industry is less than 30%. More than 70% of the projects fail. Typically, software projects fail either in terms of budget overrun, scope creep, or time overrun, or they suffer from quality issues, right. So every release, they leave behind a huge backlog of bugs. Definitely not a great situation for an industry with so much potential to impact all other industry. Right? |
| Shubham Agarwal : | Correct, but then why has the track record in the software industry been so poor? You know, why has it been so? |
| Anantha Keerthi : | So, if you rewind 20 years back, say two decades back, okay, before agile was introduced, the situation was no different, the delivery performance was not as good. At that point of time, many believed that waterfall model of software development as the major reason for such poor project performance, okay, so few in software community felt that waterfall methodology is borrowed from traditional industry dealing with physical products, you see, and therefore, this may not work for software products at all. So most of the failures were attached to waterfall model then. |
| Shubham Agarwal : | Okay, so can we look at what was the problem with the waterfall methodology then? |
| Anantha Keerthi : | Yeah, see, so if you talk to the industry practitioners, okay. And if you ask the same question, the typical answer that you get is what’s what’s what is the problem with waterfall methodology they will always say, See waterfall model is very rigid, it’s sequential in nature, you don’t get to see working software until late in the development cycle, there is no there is no flexibility okay, you cannot accommodate changing requirements right. Also waterfall approach is considered as heavy on process in terms of a lot of documentation, so sign offs, you know, bit bureaucratic in nature. So, so, so, these are some of the typical complaints against waterfall model that you come across, okay. But Shubham you must understand why waterfall model insisted so much on process adherence, right. Sure. Basically, waterfall approach require one to move the project in a sequential manner. Let’s say you start from requirement detailing then only you go to the design only after completing all the design, you move to the development. So then finally, you do the testing, right. So, if you carefully observe waterfall model tries to meet two basic objectives, okay, right. So, one is to ensure there is a clear accountability between the supplier one who is developing the product and there is a customer the one who is using it, right, right. The second objective I would say is to ensure the entire development process itself is stable. That means the level of rework is very, very low, there are no much surprises. So these are the two objectives what waterfall model is trying to meet? So for the first objective, okay, so freezing requirements before starting the design is very, very important. You know, why? |
| Shubham Agarwal : | It may, what I can think is, you know, once the requirement is frozen, you can trace back as to you know, who is accountable for any effort or the time overruns, whether it is developer or the customer, so we have a clear accountability, I would say. |
| Anantha Keerthi : | That’s right. So it means for the second objective of ensuring minimum rework, you must first freeze the design, right before starting the development of any product. Right. So this makes a lot of sense, when you’re creating good conventional, or physical products. Okay, let me give you a much better example, which, you know all the audience can understand one of them is supposed to if you’re building. Let’s take the example of in a building construction project. Right. Okay. The design changes would not be permitted while civil construction is going on. Right. |
| Shubham Agarwal : | Yeah, correct |
| Anantha Keerthi : | Just imagine, what happens if you keep changing the design while construction is happening? |
| Shubham Agarwal : | Yeah, no, it will create a lot of havoc. I would say if you tamper with the beams or these trusses, columns, I don’t know, that whole up the building, the amount of rework would be would be huge, enormous. And I think it would also mean a loss of resources, you know, time, capacity material, everything would would be lost. |
| Anantha Keerthi : | Correct. So it’s a very common sensical, right? Yeah, that entertaining design changes late in the development stage can create havoc. This we understand when it comes to dealing with physical products. So the traditional product development model has always insisted on very strict adherence to the sequence, right? For this reason, one of the overheads of waterfall methodology is a lot of documentation and sign off. That’s how they were resorting to a lot of documentation and sign offs. But however, what used to happen Is that when customer testing phase comes in the fag end when all the sequential work is over and now you’re with the customer testing phase and during this fag end a lot of surprises used to emerge right in terms of clarity issues between customer and developer, right huge gap between what was told and what was understood right a lot of scope rework detected too late in the process. Okay, all this leads to very uncontrolled delays and stress towards the end of the product release so the hallmark of waterfall was to more towards in the end a lot of surprises |
| Shubham Agarwal : | Okay, so I think the problem is well understood by now and I’m sure our listeners also do follow up, but Anantha has agile not solved this problem of the waterfall method. |
| Anantha Keerthi : | See, unlike waterfall methodology, where you don’t get to see your working software until late in the development cycle, the agile framework takes a different approach See, agile believes in delivering small but usable features in a very short interval which are just good enough for customers to start using it and then keep improving it based on the frequent feedback from the customer. See the whole idea here is to deliver workable software as soon as possible and get feedback for further iterative development. So that means there is no need to freeze the entire requirement are designed upfront, before you start doing the development. |
| Shubham Agarwal : | Okay, so but then But then does that mean that you know, the Agile Model believes that the scope and design cannot be frozen upfront in a software environment? Is that the assumption there. |
| Anantha Keerthi : | Yes, that’s right. Yeah, yeah, yeah, that’s right. See fluid requirements and design okay. This is a very important assumption behind agile framework they believed in you know fluid requirement and design. So in the in the Agile world one can move away from sequential working between phases of software development see, but if you’re going to keep everything fluid then another question comes, how are you going to execute in real world when are you going to you know stop your iteration like it’s an iterative development fine, but when are you going to stop it right. So now comes another important assumption right of agile that is self organising team. So as per agile philosophy, the team must work as a self organising group. So this self sufficient team works in a very cohesive manner. So which can concurrently do the design also, development also, testing, deliver, workable feature continuously in an iterative manner. So self organising team doing iterative development is the key factor in agile in an agile world. |
| Shubham Agarwal : | Okay, so in a sense, the team is also iterating their ways of working but then Anantha, you know, if people are working in such an iterative style, how are they going to manage the time because I think that can become a challenge. |
| Anantha Keerthi : | It’s a very good question. So, so well when it comes to time, managing time, there is another popular concept right? When it comes to agile, okay, always Agile has another concept called as sprints right. Time is usually managed using the concept of Sprints, you have heard of this? |
| Shubham Agarwal : | Yeah, yeah i have heard of this. |
| Anantha Keerthi : | So it’s a it’s nothing but a time boxing of either two weeks. In some environments, it could be four weeks. It’s a standard time boxing technique. This standard time boxing is used to deliver small batches of usable software features. So let’s get back to now your basic question, has agile improved the success rate of software projects or not? |
| Shubham Agarwal : | Yeah let’s let’s try to answer that yeah. |
| Anantha Keerthi : | Yeah. So, when agile methodology was evolved many believed that it is going to drastically change everything for good. But you know nothing like that happened even more than two decades the project performance has remained as bad as it before, except some you know improvement in one parameter but deterioration and else you know some other parameter okay when you interact with developers, you realise that the stress level has only gone up okay than before Okay, despite you know, many years of agile implementation, there is a huge backlog of bugs, customer dissatisfaction related to usability of software, right, except some except for some minor improvement. |
| Shubham Agarwal : | So, then, I would rather pose this question as to what are the limitations of agile, is it more like a conceptual problem or is the problem is in the implementation phase? |
| Anantha Keerthi : | Yeah, so, look, if someone has already implemented agile for a long period of time, but are not reaping the expected results, you, you may see a counter argument from the proponents of agile. Okay, people basically who are practitioners of agile, they may justify pointing out implementation gaps as the main reason for not getting the results and not the conceptual gap. |
| Shubham Agarwal : | Okay, so, again, how do we validate some of these arguments, then? |
| Anantha Keerthi : | Whether it’s a conceptual gap or an implementation problem, right? |
| Shubham Agarwal : | Yeah, correct, correct. Yeah. |
| Anantha Keerthi : | See, you take any theory or you take any methodology or any framework, right? Fundamentally, they operate based on certain core assumptions or certain boundary conditions within which it will work. Right, right. Right. So the first step is to surface these assumptions and check if they are violated in the majority of the cases, if these assumptions are violated in the real world scenario, or it doesn’t make any practical sense, then there is no need to check the implementation gaps at all. It’s clearly a conceptual gap in the methodology itself. So in fact, we just discussed about a few inherent assumptions of agile framework right. |
| Shubham Agarwal : | Okay. So let me quickly try to summarise it for all our listeners as well. We discussed about three important things The first is a self organising team. The second is a fluid requirement and design where the scope keeps on iterating and the third are sprints now how do we challenge these assumptions Anantha? Why do you claim that these assumptions are flawed? |
| Anantha Keerthi : | Okay, so let’s let’s take the first assumption, you talked about self organising team right. The agile framework comes from the premise that the best architectures requirements and designs emerge emerge from the self organising teams see it okay. For a team to self organise, it needs to be self sufficient first right. It cannot self organise without adequate skill sets of all required types, especially high end skills like it could be designing, it could be architecture, it could be requirement assessment, properly understanding the customer requirement elaborating, right, understanding the limitation of the product, right. While generic skills, no skill resources like developers and testers, they’re already available in the market they are readily available, okay, but subject matter experts like SMEs are always in shortage See, so, so, when agile interventions begins these experts SMEs or who are mostly groomed within the company are further shared among these self organising teams, agile teams, so even though you know you know, on the face of it, each agile team may seem adequate in terms of number of resources, they may not be self sufficient in true sense. See, lacking key skills like designing, architecture, requirement assessment therefore, the level of self organising team according to me, often is incorrect or at times misleading, |
| Shubham Agarwal : | Right, I think I think we understand and this is, this could be a, you know, reality and I can imagine how it how it shapes up. But what about the the next one the fluid requirement and design? What’s wrong with that assumption? |
| Anantha Keerthi : | Well, when, you know teams adopt agile basically what they do they do away with formal process of handovers and completion of requirements and off design phases, right? design phases, right. Earlier there was a formal handover was there, right? So, this is completely discarded now, they also dispense with traditional documentation, process and tools that were used to confirm proper requirement and design. See, so, because this was something a part of waterfall methodology where we keep complaining that it is process heavy, right? |
| Shubham Agarwal : | The processes had to be yeah |
| Anantha Keerthi : | So the underlying assumption is requirements and design cannot be frozen. So, these are left to the judgement of self organising, you know, agile team. So, now, let’s look at the paradox here. You know, what’s happening here? On one hand, agile teams are short of required skills, right? Yeah, the assumption one that we talked about. On the other hand, the formal process, checks and tools are discarded. The formal process to confirm design and requirement, everything is discarded. So what do you expect to happen in this situation? |
| Shubham Agarwal : | No, undoubtedly, I think it’ll lead to a deterioration of the design and, you know, requirement assessment, I think, |
| Anantha Keerthi : | Yes, yes. Can’t see what happens is the testing becomes the primary source of feedback on design and development gaps. This is a very, very dangerous situation, right? So there is a continuous flow back of you know testing, it causes frequent interruptions in the regular development. Right, you have to wait for the testing result to confirm your requirement and design. Literally, this is what exactly is happening there. |
| Shubham Agarwal : | Right. And I’m sure you know, leaving out the comprehensive design analysis to save time actually proves to be a much costlier proposition in the longer run. |
| Anantha Keerthi : | In the long run correct. You’re right. So initially, we felt that there is no need of doing documentation, there is no need of handover all this team will work together and like a rugby team, right. So you know, more of euphoric decision that yes, they will do everything and but you don’t have enough SMEs with you right. So the teams end up spending more time on bugs, mised use cases, regressions, and all reopen cases, all the unplanned rework basically silently guzzles up the team’s capacity, particularly that of the SMEs. And at the same time, it lengthens the entire development time. |
| Shubham Agarwal : | All right. So I think we have already invalidated two assumptions that of the self organising team and the fluid requirement and design. But we have a third one as well the sprint cycle. What about the sprint cycle? |
| Anantha Keerthi : | Yeah, well see, one of the key differentiators proposed by agile was iterative software development instead of sequential development as in waterfall, right. So, iterative development involves taking feedback and making corrections. Now, iterative development by definition is the opposite of pre planning, right? |
| Shubham Agarwal : | Correct. Yeah. |
| Anantha Keerthi : | So, so, if agile is to be followed in its purest form, then teams cannot plan all the sprints in advance. But if that is so, another question is that how can they plan overall project, have the version release? So, while they acknowledge yes, you know, we need to have an overview and there is a required, because you know, you need to answer for somebody that is somebody funding this project, right. Yeah. So, the overall view is required overall planning is required for budgets and commitments. Okay. You know, you must answer this question to address this. And you know, when it comes to Agile implementation in very large organisation, in the large enterprises, literally, they modified the agile framework now. |
| Shubham Agarwal : | Okay. |
| Anantha Keerthi : | To suit this particular need, this is where the shift starts to happen. Every sprint becomes are now a deadline with pre planned and committed scope. It’s no more iterative iterative development, right? Every sprint because there’s a time boxing, right? So every sprint timeline that we just discussed about it could be two weeks, it could be four weeks, it becomes a deadline. You need to deliver something you got to deliver something, feedback on current sprint is not required to plan subsequent sprint, right? That’s how exactly the entire team is behaving now. Each self organising team is expected to independently deliver the coveted scope within the sprint deadline. Especially SMEs you know however, because everybody comes to SMEs, right, yeah. So, they face a severe capacity crunch very close to the sprint deadline, as they support developers of current sprint, and also they need to prepare for subsequent sprint. So consequently, what happens is the tasks of current sprint get priority over the preparatory task of the future sprint, it could be designe and the quantity |
| Shubham Agarwal : | Yeah, it is understandable yeah. |
| Anantha Keerthi : | So this way, development kicks off with suboptimal design, because sprints are coming, because time is elapsing so sprints are going to come right every sprint is going to come whether you have finished or you have not finished, it’s not going to stop. So now the game is all about commitments, right. So this leads to a series of reworks, like reorks in terms of bugs, regressions, reopen cases, you know mis used cases Yeah, lot of missed implicit use cases right. So it creates a kind of a snowball effect of a task spilling over from one sprint to another from one sprint to another right. So you know all the known errors we know that there are a lot of errors you know, it will move to the subsequent sprints. Okay, now, it can move from one to another one to another but still our release cycle is the same. Ultimately customer is not interested in our sprints. He’s interested in the entire end release, right? |
| Shubham Agarwal : | Yeah, you do whatever you do but give me the final product |
| Anantha Keerthi : | Yes. final product. Yes. This This generates a growing list of bugs Okay, and leads to a massive effort spiking during the month very close to the final release. Because when it when it comes to final release, I’m sure it’s a different sprint altogether, right. There is you know no sprint any more left? |
| Shubham Agarwal : | Yeah, its all work and work only. |
| Anantha Keerthi : | Yes yes. In fact, in in certain companies, they have few special sprint, okay, only, you know, to clean up this mess. Okay, of course, it’s not possible for them to clean up this mess at times. Finally, the product is released with a lot of known defects. Have you ever heard of two teams now now you need another team to maintain only defects? Because this team wants to now move to another release? |
| Shubham Agarwal : | Oh, great. It is very interesting. I think all the core assumptions of Agile Model, the Agile sprint model are flawed. At least we have clearly invalidated them in the real world scenario |
| Anantha Keerthi : | True, so. So just to summarise what I would say is that when when deadline of sprints are imposed in an environment of resource constraints, and fluid requirements, the resultant effect can be highly damaging in terms of the entire system productivity, |
| Shubham Agarwal : | Right. So I think Anantha this is very clear. And I see a clear issue. And you know, I think the understanding is also very high in terms of why the issues are happening in the software development industry, why the entire domain is going through such hard times. Even though they’re using, you know, processes like agile, and a lot of people boast of using agile, but we clearly see the issues that are there with this model. So my obvious next question would be what is the way out? Or what is the new approach? How do we solve this, all these assumptions, all these issues, and go towards a sustainable model. But what we’ll do is we’ll break here, and we’ll come back in another episode, and discuss more on the solution part. What how can we solve the issues that creep up because of Agile? And how do we go to a more sustainable future? Anything you want to close with Anantha? |
| Anantha Keerthi : | That’s all I think, I think you have nicely summarised it. Okay, so, let’s, let’s now think of how do we it’s not that agile framework is bad, it’s not that waterfall framework is bad. It’s just that you know, like having theory of constraints that we have learned, you need to whether you know, whichever solution framework you take, okay, understand its boundary conditions, okay. Once you know the boundary conditions, then you know, right, how to you know, come out with a much better solution, right. Now. Let’s see, when we discuss about solution, more interesting thing is that there are no certain key core. I would say beliefs of a waterfall methodology, it’s really good. Similarly, there are certain major paradigm changes that are done by agile also, right, okay, you need to understand what are those? Right? How do we combine these two, both and yet, you know, using both we come out with a fantastic solution and which will, you know resolve all the current issues, right, it will touch upon all the current undesirable effects that we are facing, and how do we come out of it? Right. So, that’s the key. Right. |
| Shubham Agarwal : | Wonderful, great so looking forward to that discussion with you Anantha. For all the listeners, if you have any concerns, any questions or doubtson what we discussed today. You can write to us on our social media handles, the link is in the details. We will come back and discuss about the solution like we said. Thankyou Anantha once again, this is Shubham signing off, bye bye |
| Anantha Keerthi : | Thank you bye bye. |
Comments
Related Information
Get in touch
Registered Office:
Vector Management Consulting Pvt. Ltd.
10th floor, Thane One, DIL Complex,
Ghodbunder Road, Majiwada,
Thane (West), Maharashtra - 400610, India.
Vector Management Consulting Pvt. Ltd.
10th floor, Thane One, DIL Complex,
Ghodbunder Road, Majiwada,
Thane (West), Maharashtra - 400610, India.