How Agile Fails in Practice

I think it’s safe to say that every developer has heard of Agile or used it at some point in time in their career. I think that many people, particularly project managers and non-developers, see Agile as a blanket solution to apply to a pretty complicated problem: How do you manage a software project? Throughout the course of this article, I’m going to describe the promises Agile makes us and how it also breaks those promises.

First, I would like to start by describing the problem in more detail. In the modern world of software development, there are a number of challenges to overcome when working on a software project as a team and there are a number of factors involved in the answer. The size/scale of the project, the size of the team, the time allotted to get a product ready for market, and the software stack you’re using just to name a few. Before anything can get going there needs to be some kind of standard that the team must adhere to that everyone agrees upon. That standard has to be flexible because your requirements will change throughout the course of your project. This causes the scope of the project to fluctuate and if your team can’t handle sudden change it can throw everything off, people start to make mistakes, and things can get a little hectic. On the flip side if you don’t keep up the momentum of the project then you will run out of work, people will get bored, and mistakes will be made that could have otherwise been avoided. Both scenarios are equally damaging to your project and this is something that needs to be taken into account by your set standard.

We all do it, can’t get around it and you just have to deal with it

The last thing that needs to be taken into account is that mistakes will, inevitably, be made even if your conditions are perfect, as we’re all human and it happens from time to time. Now you have to go back from time to time and audit your code to make sure you clean up any mistakes that have been made. This is a constant factor that will need to be taken into account by the standard your team sets.

So it seems like there are a ton of problems to take into account, but the underlying cause is change. You can’t move forward if you aren’t changing, and if you aren’t moving forward you’re not accomplishing your goals. So the main issue that your standard needs to deal with is the constant change.

It’s so nice and pretty. It looks like it will work brilliantly, but it falls short in practice

Before we get into Agile I want to make sure that everyone is on the same page, so if you’re unfamiliar with Agile please read about it on the agile alliance website here. Read over the Agile 101 page, make sure you’re familiar with the manifesto, and the 12 principles.

Let’s start with what we see first, Agile 101:

Agile is the ability to create and respond to change. It is a way of dealing with, and ultimately succeeding in, an uncertain and turbulent environment.

So Agile sets out to solve the issue of constant change in your project. Great, let’s read on to the manifesto or the 4 values:

1. Individuals and interactions over processes and tools

2. Working software over comprehensive documentation

3. Customer collaboration over contract negotiation

4. Responding to change over following a plan

Let’s unpack this a little. The first value wants to place the importance of individuals who drive business needs over processes and tools. We also want to spend more time making sure that the software works well rather than documenting how it works. We want to collaborate more with our customers and continually hear their feedback rather than negotiate and adhere to a strict contract. Lastly, we want to, essentially, go with the flow instead of following a plan.

Moving on, let’s read over the 12 principles. I won’t list them all here, but I am going to talk about them. The first three principles really revolve around customer satisfaction through continuous delivery of working software within a short time period (a couple of weeks to a month at a time). The next three (4–6) focus on team structure. It says that the business people and the software developers need to work together throughout the project, we need to center our projects around motivated individuals with the support they need in order to get things done, and that we need to do more face to face meetings to convey information. The final six principles (7–12) tells us that success if working software, that sustainable development requires sponsors, developers, and users to maintain a constant pace indefinitely, that we need to self-organize, keep it simple, and lastly reflect on what we’ve done at regular intervals.

Ok, by now I can see some obvious things that are contradictory and some things that obviously won’t work in practice, but I’ll talk about my assessment later. Right now let’s talk about how most companies practice these values and principles in practice.

Agile in practice

Software teams have several tools and processes that are pretty standard for practicing Agile: Jira(come on we’ve all used it), sprints, stories, story points, stand-up meetings, sprint planning meetings, backlog refinement meetings, sprint review meetings, sprint retrospective meetings, burndown meetings, scrum, scrum masters, test-driven development (or TDD for short), and plenty of other things I won’t list here for the sake of brevity.

What exactly is all this stuff though? Well, Jira is a tool that helps you manage most of that other stuff. Jira allows you to plan your sprints, create stories for your backlog, assign story points to your stories, assign stories to your developers, and helps drive all the meetings that I listed as well. This is what is usually known as Scrum (which I also listed) that is a process used in Agile development. We know from the Agile 101 page that agile is not all about scrum, but rather an umbrella term that simply incorporates scrum, TDD, and the listed meetings.

Let’s dive a little deeper and look at these things as a whole and how they all work together under the umbrella of Agile.

Jira

Let’s start with Jira. In Jira, we can create these things called stories. Stories are just tasks with a different name and a little panache. There are some different kinds of stories, for example, a user story looks like this: “As a user, I want to be able to sign up for an account”. Most stories I’ve worked on don’t necessarily sound like that, they’re more along the lines of “Construct sign up page with…” and go on to give some more technical details and maybe attached documentation. Then there are stories for bugs, which get a little bit more technical. Stories that you cannot complete yet are put into a backlog of work to be completed sometime in the future once certain requirements are met. From there, stories are organized into sprints which are 2–4 week in length (depending on your organization). A sprint is really just a timeframe with which you are to complete a list of tasks. Now, this starts to get into some of the contradictions and where this all starts to break down.

The Agile principles tell us that success is measured by working software, but that’s not entirely true. You see all stories have points assigned to them, which are completely arbitrary values taken from the Fibonacci sequence that show the value of a certain story. The numbers MUST be from the Fibonacci sequence but their value can be arbitrary. Story points can also be used to dictate how long a particular task is going to take, but the value of a single point is really up to the organization/team to decide. I’ve seen it where a single point denotes a day’s worth of work and others where a 3 points story is a day’s worth of work. The points are tallied up for each sprint so you know how large a certain sprint is. Now one of the good things I will say about this is that it can help you estimate how large your sprints are going to be and it can also help you keep sprints to a certain amount of work.

Meetings, meetings, and more meetings

This kind of exemplifies some of the issues I’ll talk about in a bit

You’ll probably notice that my list was full of different kinds of meetings, and what are those really? Let’s start with the stand-up meeting. This is the shortest of the meetings and the one you’ll do most often. This meeting involves all team members standing over a board of some kind (usually a Jira board that has your current sprint and all the stories being worked on) and each team member will say what they did yesterday, what they are doing today, and tell the team anything they cannot work on due to missing requirements or whatever else that would be blocking them. This usually takes anywhere from 15–20 minutes depending on how busy things are. Standup meetings can also include a burndown meeting in which you go over a chart of the points that have been completed vs the points that are still left to do.

The next most common meeting is the sprint planning meeting, which can also be called a grooming meeting or an estimation meeting. Before every sprint, all the developers on your team will spend 45 minutes to an hour to plan the next sprint. During the course of the meeting team member will move tasks from the backlog, if they can, decide on a point value for each task in the next sprint, perhaps create new tasks that need to be done immediately, and assign a priority to the tasks if needed.

You also have the backlog refinement meeting where each team member will, again, take about an hour or so of their time to look over all the stories in the backlog and see which ones can be moved into sprints and which ones still need requirements fulfilled.

Now that your 2–4-week sprint is complete you have 2 more meetings you have to do. The first one is the sprint review meeting, where you will tell everyone what you did that sprint and asses if your goals were reached. The next one is the retrospective meeting. This meeting typically takes around 30–45 minutes (or more depending on how things went that sprint ) and involves each team member telling the team something they thought went well and some things they thought did not go well. You can also mention things that your team should continue to do. This information is all taken down with whatever tool your team uses (google drive, ms office, libre office, etc.) and then stored somewhere so that all team members have access to the document.

Scrum

Oooohhh it even has its own fancy chart

There were some other things I mentioned like scrum, scrum masters, and TDD. I am planning a whole other article on TDD, so be on the lookout for that one. It’ll be a doozy and not everyone will agree with me, but it will be important.

For now, we’ll focus more on scrum and scrum process. Now technically what I described above is mostly scrum which falls under the umbrella of Agile as an “Agile process”. If you want to read more about scrum, you can do so here.

As I mentioned, most of what I described above with Jira and meetings is the scrum framework. Now in scrum, you have these people called scrum masters. Scrum masters are usually (not always) non-technical project managers who are responsible for the team following Agile practices. These people will usually be responsible for all of the meetings and creating the stories and all the other non-technical legwork of the project management process.

Let’s break it down

Now that we’ve seen what Agile typically involves, let’s see where it all breaks down in practice. If you’ve been following along you can probably already pick out some things I’ll mention.

Let’s go back to the title of the article “How Agile Fails in Practice”. Agile has made several promises that we talked about, the first one is “Individuals and interactions over processes and tools”. This is provably false because the first thing you do in Agile is scrum which puts the scrum process over everything and the scrum master is to be regarded as Agile Jesus. I’ve worked on many teams where developers were strictly prohibited from creating stories, assigning points to stories, or retrieve work from the backlog/next sprint (because you know, we ran out of work for this sprint) without first meeting with the scrum master. If the scrum master happened to be busy then you can twiddle your thumbs until they’re not.

This process also nullifies a couple of the Agile principles (namely 4 and 5). These principles want the development team and the business people to communicate daily throughout the project and building projects around motivated individuals and giving them needed support. First off, communicating with the business daily is simply not realistic and can waste a lot of people’s time especially on days when there is nothing to support. Secondly, developers rarely communicate with business people because they have their Agile Jesus to do it for them. The scrum master also usually gather the business requirements and relays it to the team. This muddies the waters of communication within the business.

Let’s talk about value number 2 “Working software over comprehensive documentation”. C’mon you guys know exactly what happens in this scenario: no documentation is written. So you have a codebase with no documentation and now you want a new team member. Now we have nullified principle number 8: “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely”. This seems a little nonsensical because having your developers, sponsors, and users maintain a constant pace indefinitely doesn’t sound right, because it’s not. A constant pace is never a reality when dealing with anything because the pace of things ALWAYS fluctuates. How, exactly, does not documenting your work promote sustainable development? I gave the situation of a new team member coming into a project with no documentation, how is that handled? One answer is to have another team member take them through the codebase, but now you’re losing an effective team member who could be writing code. Another answer is to throw the new guy right into the fire and tell them to figure it out because fuck the new guy anyway. All this does is it makes it harder for your team to onboard new members, which does not help when you need to get shit done.

Speaking of sustainability is a good segue into talking about value number 3 “Customer collaboration over contract negotiation”. While this may be good for the business guys, this is an utter nightmare for every developer. Often times a customer doesn’t know what they want until they see it. So they’ll give you a set of requirement which you’ll meet, then they’ll flip the whole thing on your head and tell you it’s just “not what we envisioned”. Well fuck, now we have to refactor, go back to the drawing board and figure it out AGAIN. I’ve been on more than one project where this was a harsh reality, but to be fair, not all customers are that bad. However, even if a customer doesn’t want to change the whole thing on you, they will always change some requirements on you because they changed their minds on something or whatever other reason they give you.

Customers not knowing what they want and changing requirements is as old as time itself, but if you follow Agile, this can be way more of a pain in the ass than it needs to be. You see, the sprint structure that is set up for you is also your undoing in this case. Since your sprints are only supposed to be 2–4 weeks in length and there are these story points that determine the size and length of each sprint adapting can be really difficult, especially when your sprint is full up. So you can throw it in the backlog and get to it another time or in the next sprint, but then the customer is probably not going to be happy, and Agile principle number 1 says to always make the customer happy by delivering working software. However, principle number 1 takes away from the agile values of putting people ahead of practices and principles like sustainable development.

This brings us to value number 4 “Responding to change over following a plan”. As I was alluding to, Agile is horrible at responding to change because of the sprint structure and scrum process. How can you respond to change when Agile Jesus is getting pissed off because developers are creating tickets without permission? The sprint structure can also have the inverse effect of having little to no work. Sometimes teams try to get more members to spread the load of work, but then the work dries up and you have a bunch of people sitting around and twiddling their thumbs.

Let’s talk about all those meetings. That’s one of the principles that they really like to uphold (number 6): “The most efficient and effective method of conveying information to and within a development team is a face-to-face conversation.” There is no shortage of meetings in Agile. From stand-up to sprint review, to retrospective, you can have up to 3 or 4 hours of meetings per week. Now I know you’re thinking that it doesn’t sound that bad, but think about it like this. If you have 10 people on your team and each one of them is spending 3 hours a week in meetings you’re wasting 30 hours per week of developers not writing code. That’s around 1,500 hours per year of not writing code. In the interest of fairness, this is not always the case, but it is the case a lot of the time. This takes away from time you need to “Deliver working software frequently” (principle number 3).

I also have a serious bone to pick with story estimation. What kind of fresh hell have I been thrown into where Agile Jesus’ head explodes if a story has 9 points. Why do story points HAVE to come from the Fibonacci sequence you may ask? Right here: “It’s better to be roughly right than precisely wrong”. So basically the value you give is meant to be a very rough estimate. Any time I see language like that it pings my bullshit radar. Roughly right, doesn’t actually mean you’re right either. I understand that this was meant to overcome the challenge of estimating the time it will take to complete something, which can be difficult for a lot of developers. However, I think this falls short of the mark. This is just another way that value 1 is disrespected because if you suggest a story may have 4 points Agile Jesus pops an aneurysm. The other issue I have is that I thought working software was supposed to measure success? However, we have teams measuring success by the stories they’re completing and how many points have been done. I think in this case the point gets a little skewed from dear old principle number 7.

I’ve pointed out that Agile fails all of the 4 values and many of their principles, but do they hold up their end of the bargain on anything? Kinda. Principle 6 and 12 (value of face to face meetings, and performing retrospectives) are done a shit load. As I’ve mentioned, this doesn’t really help developers at all.

What’s the point?

Agile is sold to developers to help make software faster and better and cooler. However, I have only ever seen Agile backfire on developers and has helped make our jobs harder and more inundated with process than is necessary. Agile should not be seen as the friend of a developer or a tool in our belt, but as a weight tied around our ankles before we’re tossed into the middle of the ocean.

Agile exists for businesses to assert their control over developers and everything developers do. Many companies use it as a way to strip developers of creative freedom or any kind of autonomy. It is a way to hold developers back from actually achieving anything by forcing us into unnecessary meetings instead of writing code.

Is every place going to be as horrible as I just described? Absolutely not, but there are altogether too many places where my description only scratches the surface.

What’s the answer?

I don’t think we’re ever going to find a blanket solution to project management. The reality of it is that, for many companies, Agile may be just fine. Perhaps they’ve decided to throw out the rule book and do it their own way that works for them. I applaud the effort of companies to think outside the Agile box and actually innovate. Perhaps Agile out the box is working just fine for the time being, and that’s fine too.

However, I think that Agile does serve the business more than it serves developers. I think this is counter-intuitive to running a software company because developers are literally the only thing holding up your business. If every business lost all of its developers they would all be completely screwed. I don’t think I can say the same of salespeople, business people, or any other organization inside a software company (I think that we can even say this of many companies that are not strictly software companies).

I think that, as developers, we need to push for something that respects us and our time more than Agile currently affords. I don’t think we need scrum masters or scrum at all. I don’t think that doing standup EVERY DAY. I don’t think we need to spend so much time planning sprints and assigning story points.

I believe that a software project is a living, breathing thing that is given life by developers and I think software projects should be treated as such. I think that we should invest time in trying to document our code, whether that be inline comments or markdown docs in the repo. I think that meetings should happen asynchronously in real-time. What the fuck does that mean? With tools like Slack and Microsoft Teams, it has become easier than ever to meet with your team in a way that does not require everyone to stop everything and go to a separate location. I understand if people want to do that, sometimes we need to leave our computers (even if I don’t want to) and that’s quite healthy.

Basically, I want us to be able to transition to a more developer-centric methodology. We need to be able to take back control of the development process and make it our own. I’m currently working on a loose set of principles that may help us do just that, but that will be covered in another article.