Agile

How Not to Succeed with Agile

Scott S NelsonLeave a comment

Originally published at developer.com

Why Agile Projects Fail

The purposely provocative title is meant to capture the attention of people with two opposing mindsets. The first are the proponents of agile methods who will want to pick apart any arguments against their preferred approach. The second are those who oppose agile and are looking for justification for their concerns. The goal of this article is to reduce the opposition between these two schools of thought, and to provide some food for thought to those who are undecided.

The truth is development projects following agile methodologies have failed. Projects following waterfall and RUP have also failed. Just as the cliché that one should not judge a book by its cover is often true (I missed out on the intellectual pleasure of Robert Heinlein’s Friday for many years due to that primitive prejudice), one should also avoid drawing a conclusion about a topic based on the title alone (The Art of War is more about how not to fight).

We will not go into deep detail of agile methodologies today. There are too many to cover adequately in a single reading, and the specifics of these methodologies do not have a direct impact on why agile projects succeed or fail. Agile is as much a way of thinking about projects as it is a set of tools that can be applied to projects, and the aspects that make up successful thinking can apply to agile and non-agile methodologies.

The Project

Both definitions of agile in one online dictionary include the word “quick”. The word agile evokes thoughts of athletic prowess and graceful, fast feline predators. It is not surprising the people who chose to adopt agile methodologies for the first time do so when facing a tight deadline. Perhaps choosing a methodology by its name will be the new cliché.

People who have followed agile methodologies for multiple projects will tell you that they provide quick results, and can show you artifacts that prove it.What they may have forgotten (or simply neglected to mention) is that successful agile projects have a few advantages to be successful with agile methodologies.

For an IT shops’ first agile project to be successful, the project must be the right project for that shop to adopt agile techniques. The project should be small, in both time frame and team size. The managers of the project must be willing to either adopt agile deliverables as they are, or adapt their method of measurement. Most importantly, the project team either needs to have enough members already experienced in agile or be given the latitude to get through the learning curve.

Agile methodologies rely on continuous and open communication between all levels of stake holders.For traditional waterfall groups, it is unheard of for a developer to contact a business sponsor or client to find out what is meant by a particular requirement description. In an agile project, the ability to do so can mean the difference between success and failure.

Agile methodologies can be used successfully to deliver a large project, but not by a team that is learning to use these techniques for the first time. Large projects often include many team members, at multiple locations. An IT shop can grow a small agile team to a large, distributed team over a few projects, but it is likely to be disappointing to try to assemble a large team from scratch, even if all of the members are experience in agile. Like many adjectives, agile has different meanings to different people in different contexts, and two highly experience agile project members could have entirely opposing approaches. Large agile teams must be grown organically rather than assembled randomly. These caveats to applying agile projects are most likely the root of the general consensus that agile is not appropriate for large projects. Perhaps this caution is a good idea when introducing agile methodologies to a team for the first time.

In describing the ingredients of projects that can succeed with agile above, it is important to note that the different modal operators of probability used are chosen specifically, rather than simply to provide variety in wording. Project managers new to agile must be willing to adapt. Teams new to the agile approach as a team needs to either have enough members with agile experienced or be given the opportunity to stumble and learn. Communication across stake holders can impact results. There are exceptions, such as when those with roles that include being the translator of business requirements to technical requirements have good relations and communications with business and development and are highly competent in those communications.

This section began with a reference to a dictionary definition. In addition to the current English meaning of the word, most dictionaries include the history of a word, its etymology. At that same reference link, the etymology of agile is given as “from Latin agilis, from agere to drive, act”. So, apparently the meaning of agile in the English language evolved from its origins to include the notion of “quick”. Agile methodologies can evolve within an organization to be what we want them to be, though it is unlikely that they can be exactly what is desired the first time out.

The People

The introduction mentioned that there are opposing mindsets about agile. It also included a reference to The Art of War, and these are not coincidences. An agile project team with members of both camps has two strikes against it from the start. It only takes one more strike to be out.

Many proponents of agile tend to be over zealous in their belief of the superiority of their preferred agile methodologies. While their enthusiasm may win support from the undecided, it can also cause those opposed to agile methodologies to push back even harder. It can also frighten those who are participating in an agile project for the first time, as agile methodologies are very different from waterfall approaches and change causes anxiety for many. Where patient mentoring will frequently overcome FUD (Fear, Uncertainty and Doubt), ignoring the FUD of others generally results in greater FUD.

Most opponents of agile are against it because of FUD. Sometimes rightly so, as agile is not a panacea and is not the perfect methodology for every combination of projects and teams. Those most opposed to agile methodologies are non-developer team members. Experienced waterfall project managers and analysts are highly trained in creating complete definitions of deliverables and measuring progress towards those deliverables with specific tools. While agile is still a measurable methodology, the measurements are defined as the project progresses instead of up front. This can be too far outside of the comfort zone of some people. And people who are uncomfortable will often do whatever it takes to become comfortable again. For some, that is achieved by adapting and learning, i.e., becoming agile themselves. For others, the approach is to make every effort to bring things back to the way they are used to. These latter individuals most likely do not do it consciously, but they will sabotage an agile project. How? By doing exactly as they are trained.By trying to capture every minute detail for a story definition rather than outlining it and providing feedback during the daily testing. By trying to fit too many features into a single iteration. By thinking a milestone was missed because an iteration did not deliver all of the features planned.By not delivering requirements that are completely defined because the due date for the requirement has not yet arrived on the calendar. By running an agile project in a waterfall mentality. In short, by not being part of the agile process by being agile themselves.

To repeat, those who oppose agile processes and sabotage agile projects most likely do not do it consciously. If everyone isn’t onboard with the agile process at the beginning of the project, it can still succeed. If everyone isn’t onboard with agile by the end of the project, that project will most likely have been a failure. If everyone pulls their own weight, but pull in a different direction, something is bound to come apart.

Those who have delivered successful agile projects have good reason to be confident in their approach. Successful agile projects result in solid software, often delivered either under-budget or with more features than originally projected. The iterative approach of providing rapid, demonstrable deliverables quickly builds enthusiasm for both the clients and the developers. Those who began the agile project with FUD and then learned to adapt and adopt new processes are those who become the biggest supporters of agile. It is important that they remember that they had their doubts at first if they are to convince others who suffer FUD that they, too, can become successful agile project members.

Hopefully you realize that this section has many redundancies to the first section. Project deliverables may be software and documentation, but it is people who deliver them.

The Process

Waterfall projects are successful when there is a rigid adherence to process. To some, that rigid adherence may be the only ingredient they are aware of. They may not acknowledge that the project also required the experience and knowledge to create an accurate project plan with achievable milestones.

In agile projects, rigid adherence to process is a key ingredient to failure. Agile is made up of many different processes that can be adopted, combined, added, dropped and modified as needed. First-time agile projects are prone to the mistake of choosing their processes and sticking to them even when they don’t work.

A perfect example is Test Driven Development (TDD). In TDD, automated test procedures are written first, then the development is done and tested continuously. This is a great approach to ensure functional quality. It works perfect for the Control layer of an MVC architecture, for example. However, for the view layer of a web project, it can result in deliverable falling out of synch as it takes much longer to write tests for web pages than it does to write the page itself.

Daily stand up meetings are an excellent communication tool. Run properly, they provide an excellent benefit. However, if one or more of the unconscious saboteurs join in (or, as is often the case, lead) these meetings, they can become a key factor in iteration failure. Choosing to make the stand up every other day or weekly while leveraging instant messaging can put an iteration back on track if the daily stand up is detracting from productivity.

Another fallacy held by extreme supporters and detractors of agile is that it cannot be combined with waterfall. Nothing could be further from the truth. An agile development team that begins prototyping based on the initial, high-level requirements of a waterfall project will be able to demonstrate their achievements earlier in the development stage and get immediate feedback from clients. Because another fallacy about waterfall is that all of the requirements are defined before development begins. Change orders are part of the waterfall process, and it is very rare they aren’t used. The earlier the change order occurs, the better the chances of success are, and agile development processes within a waterfall project facilitate these early changes in direction that lead to improved deliverables.

Agile processes work if implemented in an agile manner and followed by people willing to be agile.

Conclusion

Why do agile projects fail? When agile methodologies are not used in a project labeled as agile. Whether you consider the root of the word agile as meaning to take action and get immediate results, or the modern meaning of quick deliverables, simply using the word agile is not enough to take action or be quick about it. Nor is simply taking quick action truly agile, unless it is done with purpose and support. Many of the most successful agile projects I have participated in were not labeled agile; we simply followed agile methodologies as a development team and met waterfall deliverables, usually ahead of schedule.

Agile projects do succeed, and agile methodologies are designed for success. Like any tool, if it is not used properly it will not accomplish what it is designed for, and can cause actual damage. If you have a fly to swat, a fly swatter works well… unless it is tied to a Buick.

Additional Reference Links

The Wikipedia entry on Agile at http://en.wikipedia.org/wiki/Agile_software_development contains many disclaimers, making it more honest than most references found during the writing of this article.

A short list of SDLC approaches can be found at http://testdog.com/knowhow/Sorting%20Out%20SDLC.html. While there are many, many other sites about various SDLC approaches, this one seemed to have the least agenda and the most variety.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

The Art of Stubbing

Scott S NelsonLeave a comment

(Originally published as Stubbing For Fun, Profit, and Survival )

Anyone reading or watching the news these days know that IT project budgets will be tightening further than you may already be experiencing. While proponents of some languages and platforms claim to provide lower development costs, many others firmly believe that it is the habits of the developers who prefer those environments (more than the environment itself) that leads to rapid results. One of these habits is the proper use of method stubs. By proper, let’s define that as stubs that facilitate the development process first, and checking off “done” on a project plan second.

Stubbing Defined

Half the time someone asks me if I am the person who wrote a particular article it is a manager or analyst rather than a developer. Since the next person who asks you what you mean by “stubbing” may be using this article as their only reference, I thought I would add something that makes it easy to clarify this technique, and some fodder that will allow you to either argue or support my particular view on the topic…

Wikipedia defines a stub as “a piece of code used to stand in for some other programming functionality”. Sadly, the use of stubs in “general software development and testing” is thrown in as an after-thought. They give much more emphasis to its use in distributed computing, which is logical in the sense that the project failure rate is much higher if stubs aren’t used in distributed application development, and some techniques, such as SOAP-based web services require them as part of the API. In my mind, the only reason not to stub is when you can write the final code from end to end as a single developer in a single day. Any task more complex requires some type of stubbing, the specifics of which will vary between teams.

Benefits of Stubbing

The following are just a few of the advantages of developing methods stubs that return a meaningful value. I’ve left out some as obvious (despite the fact that I often say the obvious bears repeating in development) and others that are a bit inflammatory (having worked many long hours fixing the issues of not stubbing properly I find it hard to be objective).

The Ability to Demonstrate Progress on Demand

Many developers will only stub their methods if they are working in a large team. One value of stubbing methods even when working alone is that not only will the code compile, the code will run and return meaningful values. While this may appear as a minimal value to code quality and speed of development, it can be an enormous advantage to demonstrate to non-technical user (i.e., those that determine your budget) exactly how much progress you have made. While the code may be in the same stage of completion with or without meaningful return values, it is far more acceptable to clients and management to explain that the demonstration is not ready for production than to have to constantly excuse test cases that fail because of a null value.

Facilitating a Test First Development Approach

Test first development is a very useful Agile method. By writing test cases first that check for valid values and then implementing code that moves the test cases from “fail” to “pass”, you will know that your methods are written correctly at the earliest point possible in the development process. By running your test cases automatically at each build you can then pinpoint many bugs immediately after they are written rather once your code is integrated into the full project. Integrated code takes longer to debug for many reasons, one of which is that everyone assumes it is someone else’s code, thus costing the time of multiple developers to locate the issue rather than the one developer who wrote it.

Parallel Development

While it can be argued that it is possible to develop in parallel without providing stubs, it is rare that the eventual integration of layers developed in parallel goes smoothly if stubs with meaningful return values are not used to test the contracts between teams early and often.

Weekends Off

Ok, this one won’t apply to everyone, but I bring it up because during the holiday weekend I wrote this article I was also debugging integration issues for the very reason that the points covered in this article were not followed by experienced developers. I understand that it was not their intent to cause more work, which is why the lessons here are important for both the newbies and seasoned professionals.

Misconceptions about Stubbed Methods

Stubbing is Only Necessary at the Model Layer

As mentioned under Benefits of Stubbing, one advantage to stubs is the ability to demonstrate on demand. The HTML of a view object that is blank or displays “null” is far less likely to help your budget case than seeing “John Doe”, which is obviously not real data but is meaningful data.

When working in parallel development it may be tempting to point at a project plan that has dependencies being delivered in perfect synchronization as consider stubbing a controller with null because the model should be ready before the view is integrated. In the event that everyone does not hit their milestone exactly (gee, how often does that happen?) development can fall behind waiting for someone else to catch up. Alternatively, if you get ahead of your own schedule you will not be able to progress until your dependencies are at the same plan stage as you. If all developers stub their code with meaningful value at every step, they are not dependant on others until the actual point of integration, and then only in the event of bugs, which are far fewer if stubbing was occurring at each layer and each stage.

Null is Not a Stub

There are two phrases that will raise the blood pressure of many team leads and development managers. The first is “it works on my machine”, and the second is “it compiled for me, so I checked it into source control”. It isn’t the mere fact that a developer’s machine will not be the one used in production that is the issue here (though, it is one of the obvious things that still need to be pointed out often), it is that these phrases are only used when the code in question has broken the application as a whole. The most common cause of integration builds failing is a null value that should not be null. It may even (and probably will) compile within the integrated application. The failure point is when the application is run, a point that is far more costly and the a compilation failure because it can impact other developers (who have to trace their code to find the cause from someone else’s code), QA engineering teams (who have to reschedule testing because the application isn’t available), and likelihood of future team projects if this disaster occurs during a client demonstration.

The effort involved in adding and later removing:

//FIXME: Remove when data available
if(value==null) {return “John Doe”;}
else return value;

should total about 15 minutes, versus the average 1.5 man hours it takes to detect the issues caused after deployment. And 15 minutes is on the long side, as it took me 25 seconds to write the above example and should take no more than 1 minute to remove it.

“Foo” Is Only Missing an “L”

Our example above returns a meaningful value. Even in the case where full JavaDoc comments are omitted, any developer seeing “John Doe” returned has a pretty good idea how they can use the method.  A return value of “foo” would give no such indication. Even if the method has full JavaDoc notations, the value “foo” is very unlikely to be of use to the client code. It may be helpful to consider that code examples that use “foo” generally use “bar” as the second value. “foo bar” is a homonym for FUBAR, which is an acronym for “F***** Up Beyond All Recovery”.

I recently worked on a project where stubs returned both null and “foo”. These values, coupled with generated JavaDoc comments rather than informative descriptions, led to delays in integration testing and the need to re-write 70% of the consumer code once integration began due to the assumed values not matching the actual values. A valid value returned by stubs for all methods would have prevented this and saved a substantial amount money (and one of my weekends).

Meaningful Stubs May End Up In Production

Some people believe that cleaning up stubbed code takes too much of their time, or that stubs can make their way into production by being missed. Looking at that second argument, it is only valid if comments are not used properly. All modern IDEs will provide a listing of comments that start with FIXME and TODO. Stubs that are commented with FIXME and cultivating the habit of reviewing these lists will always prevent stubs from making it to production and lead to cleaner code. Using these notations also allows for being able to quickly review code where stubs have been left and remove them when they are no longer needed.

Figure 1: Task View in Eclipse
Figure 1: Task View in Eclipse

 

Using a Stub Harness

If you don’t know how long it will be until you have data, consider writing a stub harness. A stub harness is a simple check to see if the class should use stubbed values before returning them. Whether to use the stubs or not can be set in a property file at a class or as a node in your deployment descriptor. This value can be set at a method level:

if(property.thisMethod.useStubs)

at the application level:

if(property.useStubs)

or both:

if(property.useStubs && property.thisMethod.useStubs)

The key to this approach is to make sure your build process will never allow these values to be set in either pre-production or production (or other applicable environments, depending on how your organization operates). Providing your processes include a regression tests in pre-prod, methods that are still stubbed will fail prior to going to production. It also allows you to switch off the stubs at any stage of development to run integration tests.

Conclusion

There are two inspirations for my article topics. Things that are cool, and things that are painful. In both cases my goal is to share the solution in the hopes of saving others the long, tedious discovery process that I and others have gone through. In the case of this particular topic, the tedious process isn’t the discovery but the realization that an approach that may seem at first glance to be time consuming is in fact time saving.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson