Originally published at developer.com

Why Agile Projects Fail

The purposely provocative title is meant to capture the attention of people with two opposing mindsets. The first are the proponents of agile methods who will want to pick apart any arguments against their preferred approach. The second are those who oppose agile and are looking for justification for their concerns. The goal of this article is to reduce the opposition between these two schools of thought, and to provide some food for thought to those who are undecided.

The truth is development projects following agile methodologies have failed. Projects following waterfall and RUP have also failed. Just as the cliché that one should not judge a book by its cover is often true (I missed out on the intellectual pleasure of Robert Heinlein’s Friday for many years due to that primitive prejudice), one should also avoid drawing a conclusion about a topic based on the title alone (The Art of War is more about how not to fight).

We will not go into deep detail of agile methodologies today. There are too many to cover adequately in a single reading, and the specifics of these methodologies do not have a direct impact on why agile projects succeed or fail. Agile is as much a way of thinking about projects as it is a set of tools that can be applied to projects, and the aspects that make up successful thinking can apply to agile and non-agile methodologies.

The Project

Both definitions of agile in one online dictionary include the word “quick”. The word agile evokes thoughts of athletic prowess and graceful, fast feline predators. It is not surprising the people who chose to adopt agile methodologies for the first time do so when facing a tight deadline. Perhaps choosing a methodology by its name will be the new cliché.

People who have followed agile methodologies for multiple projects will tell you that they provide quick results, and can show you artifacts that prove it.What they may have forgotten (or simply neglected to mention) is that successful agile projects have a few advantages to be successful with agile methodologies.

For an IT shops’ first agile project to be successful, the project must be the right project for that shop to adopt agile techniques. The project should be small, in both time frame and team size. The managers of the project must be willing to either adopt agile deliverables as they are, or adapt their method of measurement. Most importantly, the project team either needs to have enough members already experienced in agile or be given the latitude to get through the learning curve.

Agile methodologies rely on continuous and open communication between all levels of stake holders.For traditional waterfall groups, it is unheard of for a developer to contact a business sponsor or client to find out what is meant by a particular requirement description. In an agile project, the ability to do so can mean the difference between success and failure.

Agile methodologies can be used successfully to deliver a large project, but not by a team that is learning to use these techniques for the first time. Large projects often include many team members, at multiple locations. An IT shop can grow a small agile team to a large, distributed team over a few projects, but it is likely to be disappointing to try to assemble a large team from scratch, even if all of the members are experience in agile. Like many adjectives, agile has different meanings to different people in different contexts, and two highly experience agile project members could have entirely opposing approaches. Large agile teams must be grown organically rather than assembled randomly. These caveats to applying agile projects are most likely the root of the general consensus that agile is not appropriate for large projects. Perhaps this caution is a good idea when introducing agile methodologies to a team for the first time.

In describing the ingredients of projects that can succeed with agile above, it is important to note that the different modal operators of probability used are chosen specifically, rather than simply to provide variety in wording. Project managers new to agile must be willing to adapt. Teams new to the agile approach as a team needs to either have enough members with agile experienced or be given the opportunity to stumble and learn. Communication across stake holders can impact results. There are exceptions, such as when those with roles that include being the translator of business requirements to technical requirements have good relations and communications with business and development and are highly competent in those communications.

This section began with a reference to a dictionary definition. In addition to the current English meaning of the word, most dictionaries include the history of a word, its etymology. At that same reference link, the etymology of agile is given as “from Latin agilis, from agere to drive, act”. So, apparently the meaning of agile in the English language evolved from its origins to include the notion of “quick”. Agile methodologies can evolve within an organization to be what we want them to be, though it is unlikely that they can be exactly what is desired the first time out.

The People

The introduction mentioned that there are opposing mindsets about agile. It also included a reference to The Art of War, and these are not coincidences. An agile project team with members of both camps has two strikes against it from the start. It only takes one more strike to be out.

Many proponents of agile tend to be over zealous in their belief of the superiority of their preferred agile methodologies. While their enthusiasm may win support from the undecided, it can also cause those opposed to agile methodologies to push back even harder. It can also frighten those who are participating in an agile project for the first time, as agile methodologies are very different from waterfall approaches and change causes anxiety for many. Where patient mentoring will frequently overcome FUD (Fear, Uncertainty and Doubt), ignoring the FUD of others generally results in greater FUD.

Most opponents of agile are against it because of FUD. Sometimes rightly so, as agile is not a panacea and is not the perfect methodology for every combination of projects and teams. Those most opposed to agile methodologies are non-developer team members. Experienced waterfall project managers and analysts are highly trained in creating complete definitions of deliverables and measuring progress towards those deliverables with specific tools. While agile is still a measurable methodology, the measurements are defined as the project progresses instead of up front. This can be too far outside of the comfort zone of some people. And people who are uncomfortable will often do whatever it takes to become comfortable again. For some, that is achieved by adapting and learning, i.e., becoming agile themselves. For others, the approach is to make every effort to bring things back to the way they are used to. These latter individuals most likely do not do it consciously, but they will sabotage an agile project. How? By doing exactly as they are trained.By trying to capture every minute detail for a story definition rather than outlining it and providing feedback during the daily testing. By trying to fit too many features into a single iteration. By thinking a milestone was missed because an iteration did not deliver all of the features planned.By not delivering requirements that are completely defined because the due date for the requirement has not yet arrived on the calendar. By running an agile project in a waterfall mentality. In short, by not being part of the agile process by being agile themselves.

To repeat, those who oppose agile processes and sabotage agile projects most likely do not do it consciously. If everyone isn’t onboard with the agile process at the beginning of the project, it can still succeed. If everyone isn’t onboard with agile by the end of the project, that project will most likely have been a failure. If everyone pulls their own weight, but pull in a different direction, something is bound to come apart.

Those who have delivered successful agile projects have good reason to be confident in their approach. Successful agile projects result in solid software, often delivered either under-budget or with more features than originally projected. The iterative approach of providing rapid, demonstrable deliverables quickly builds enthusiasm for both the clients and the developers. Those who began the agile project with FUD and then learned to adapt and adopt new processes are those who become the biggest supporters of agile. It is important that they remember that they had their doubts at first if they are to convince others who suffer FUD that they, too, can become successful agile project members.

Hopefully you realize that this section has many redundancies to the first section. Project deliverables may be software and documentation, but it is people who deliver them.

The Process

Waterfall projects are successful when there is a rigid adherence to process. To some, that rigid adherence may be the only ingredient they are aware of. They may not acknowledge that the project also required the experience and knowledge to create an accurate project plan with achievable milestones.

In agile projects, rigid adherence to process is a key ingredient to failure. Agile is made up of many different processes that can be adopted, combined, added, dropped and modified as needed. First-time agile projects are prone to the mistake of choosing their processes and sticking to them even when they don’t work.

A perfect example is Test Driven Development (TDD). In TDD, automated test procedures are written first, then the development is done and tested continuously. This is a great approach to ensure functional quality. It works perfect for the Control layer of an MVC architecture, for example. However, for the view layer of a web project, it can result in deliverable falling out of synch as it takes much longer to write tests for web pages than it does to write the page itself.

Daily stand up meetings are an excellent communication tool. Run properly, they provide an excellent benefit. However, if one or more of the unconscious saboteurs join in (or, as is often the case, lead) these meetings, they can become a key factor in iteration failure. Choosing to make the stand up every other day or weekly while leveraging instant messaging can put an iteration back on track if the daily stand up is detracting from productivity.

Another fallacy held by extreme supporters and detractors of agile is that it cannot be combined with waterfall. Nothing could be further from the truth. An agile development team that begins prototyping based on the initial, high-level requirements of a waterfall project will be able to demonstrate their achievements earlier in the development stage and get immediate feedback from clients. Because another fallacy about waterfall is that all of the requirements are defined before development begins. Change orders are part of the waterfall process, and it is very rare they aren’t used. The earlier the change order occurs, the better the chances of success are, and agile development processes within a waterfall project facilitate these early changes in direction that lead to improved deliverables.

Agile processes work if implemented in an agile manner and followed by people willing to be agile.

Conclusion

Why do agile projects fail? When agile methodologies are not used in a project labeled as agile. Whether you consider the root of the word agile as meaning to take action and get immediate results, or the modern meaning of quick deliverables, simply using the word agile is not enough to take action or be quick about it. Nor is simply taking quick action truly agile, unless it is done with purpose and support. Many of the most successful agile projects I have participated in were not labeled agile; we simply followed agile methodologies as a development team and met waterfall deliverables, usually ahead of schedule.

Agile projects do succeed, and agile methodologies are designed for success. Like any tool, if it is not used properly it will not accomplish what it is designed for, and can cause actual damage. If you have a fly to swat, a fly swatter works well… unless it is tied to a Buick.

Additional Reference Links

The Wikipedia entry on Agile at http://en.wikipedia.org/wiki/Agile_software_development contains many disclaimers, making it more honest than most references found during the writing of this article.

A short list of SDLC approaches can be found at http://testdog.com/knowhow/Sorting%20Out%20SDLC.html. While there are many, many other sites about various SDLC approaches, this one seemed to have the least agenda and the most variety.

© Scott S. Nelson