From Agile to Fragile in 60 sprints

The adoption of agile software development methodologies has been a necessary evolution to support the explosive demand for new and expanded capabilities.   There is no doubt that without the broad adoption of agile practices much of the growth in technology, and all of those aspects of everyday life that is driven by technology, simply would not have happened.

Still, too much of a good thing applies. Another old adage that comes to mind is “You can have it better, cheaper, faster. Pick any two”. Many organizations have insisted on all three. How did they do it? They sacrificed the documentation.

I’m not talking about saving shipping costs and trees by making manuals virtual, and then saving bandwidth by replacing the documents download with the install files with links to online documentation (which has its own issues in this world of massive M&A). I’m talking about all those wonderful references that development teams, sometimes backed by technical writers, produced so that others may pick up where they left off to maintain and enhance the final applications. Yes, that documentation.

Self-Documenting Code does not make a Self-Documenting Solution

While no one can honestly disagree with the value put forth in the Manifesto for Agile Software Development : “Working software over comprehensive documentation”, I also don’t think the intention was that documentation impedes working software.   Still, the manifesto has fed the meme (the original definition, not the funny GIFs) “Good code is self-documenting”. When I hear this, my response is “True; and knowing what code to read for a given issue or enhancement requires documentation”.  My response lacks the desired impact for two reasons: It doesn’t easily fit on a bumper sticker and it requires putting time and effort into a task that many people do not like to do.

The danger of little or no documentation is that the application becomes dependent on “tribal knowledge”. In a perfect enterprise, this is a dependable approach because employee turnover is low and when people do depart they always do so with adequate notice and thoroughly train their replacements. I have heard these enterprises exist, though I have never spent any time working with one of them.  I did, however, recently work with a business intelligence group where their entire ETL staff departed within a few weeks of each other after a few years of furiously building hundreds of data integrations in a dozen different business areas and then spent less than 9 hours in “knowledge transfer” sessions with my team who were tasked with keeping the lights on until a new crew was hired and trained. There was not one page of documentation at the start of the knowledge transfer and I have yet to find a line of documentation in any of the code.

I’m not advocating the need for waterfall-style detailed design documents. In some ways, those can be worse than no documentation because they are written before the code and configurations they are intended to describe are created and fail to be updated when the actual implementation deviates. In an agile world, writing the documentation after the implementation is a sound approach that will support the manifesto value of “Working software over comprehensive documentation” by being just enough documentation to facilitate maintaining the software in the future.

Meeting between the Lines

How much is just enough? That is going to vary by both application (and/or system) and enterprise. Some applications are so simple that documentation in the code to supplement the “self-documenting” style is sufficient. More complex solutions will need documentation to describe things from different aspects, and the number of aspects is effected by whether maintenance is done by the development team or a separate production support group. The litmus test for whether your documentation is adequate is to take a look at it from the perspective of someone who has never heard of your application and needs to be productive in maintaining or enhancing it in less than a day. If you have difficulty in adopting that point of view (many people do, and double as many developers), have someone outside your team review the documentation.

I find the following types of documents to be a minimum to ensure that a system can be properly managed once released to production:

  • Logical System Architecture
  • Physical System Architecture
  • Component Relation Diagrams
  • Deployment Procedures

Again, the level of detail and need for additional documentation is going to be driven by complexity and experience. Another factor is how common the relevant skills are. If the candidate pool for a particular platform or framework is shallow, more detail should be provided to act as springboard for people that may be learning the technology in general while diving into the particular implementation.

Yes, there are Exceptions

Conversely, some solutions are true one-offs that are filling a very specialized need that is unlikely to evolve and may have a short lifespan. These implementations only really need sufficient reference to migrate them to another environment or decommission them to free up resources while not negatively impacting other systems. I do caution you to really be sure that an application falls into this category before deciding to minimize the documentation.  What comes to my mind when I think of such decisions is massive amount of resources dedicated to dealing with two-digit years in 1999 to address applications that were not expected to still be in use when they were developed 10 or 20 years previously.

A Final Appeal

At the beginning I agreed with the manifesto value of working code prioritized over comprehensive documentations. In the days when most software life cycles began with tons of documentation and meetings to review the documents and meetings to review the results of the review, a great deal more beneficial build and test activities could have been done in that time instead. My experience in documenting the results of agile and other iterative processes toward the end of the development cycle and then reviewing that documentation with people outside the team is that design flaws are discovered when looking at the solution as a whole rather than the implications to individual stories in a sprint. The broader perspective that waterfall tried to create (and often failed since most waterfall documentation does not match the final implementation) can be achieved better, cheaper and faster when documenting at the end of the epic. In this one case, picking cheaper and faster yields better.

Documenting the fruits of your software and application implementation labors may not be the most exciting part of your team’s work, but the results of not documenting can become the most painful experience for those that follow…or your next gig!

Originally published at InfoWorld

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

The First Step of a Journey that Began Five Years Ago

Note: I will update this article with a link to the application once the customer has done their own announcements in accordance with their external communication policies and procedures.

In the Beginning there was BEA WebLogic Portal

In 2008, Oracle acquired BEA Systems. 19 days after the official merger, Oracle announced that premiere support for WebLogic Portal would end in 2014. The current policy document (latest can be found on http://www.oracle.com/us/support/lifetime-support/index.html) has moved this date out to 2018, though they have been sticking to the “no new feature release” policy since the 10.3.2 release in 2010. 10.3.2 was intended to be 11g, except it came out a year later than originally announced at Open World in 2008 and was released as a “dot” release of 10g despite the fact that it had several major enhancements and new features.

I had been hired by BEA in 2006 as a WebLogic Portal consultant due to my extensive experience with the product as a consultant for netNumina Solutions. In 2009, Oracle released WebCenter 11g and I attended the Masters Training two weeks prior to the GA date where I learned just how very different the two portal products are.

Which Way Do We Go?

I have been unable to find any officially published direction for WebLogic Portal customers who wish to migrate to WebCenter Portal, though I have had numerous conversations with engineers, architects, consultants and product managers about how to go about this. These discussions revealed three general approaches.

One approach is to simply re-build the portal in WebCenter. This is quite viable for very small portals and avoids the pitfall of other approaches, which is the need to maintain two architectures. It is not a very practical approach for medium to large portals as it is a great deal of effort and expense over a long period of time to just to provide the same functionality.

Both the second and third approaches are about transitions. On method is to create the new WebCenter portal and build all new features there and link over to the legacy WebLogic Portal for existing features. This is very quick and easy to deliver but difficult to maintain.

The third approach is staged migration. This approach creates the a new WebCenter portal that is the where users log in and interact, with the legacy functionality being exposed using WSRP. This solution allows for the immediate introduction of the WebCenter architecture and minimizes maintenance cost. By following a policy where any legacy portlet that requires modification be first moved over to WebCenter, Business and Technology stake holders can plan the complete retirement of the WebLogic Portal infrastructure as best suits the Enterprise as a whole.

Every Journey Starts with a Sprint

This month marks the deployment of the third solution to production for my current client. It is a medium-sized, high-complexity portal and it was brought from inception to production in six months using a mixed-Agile approach. It consists of 20 portlets produced by the legacy WebLogic Portal application and two legacy JSF portlets that were migrated in two days because they include file download functionality that made them easier to migrate than to dig through the documentation to fix as WSRP. The portal also includes managed content from WebCenter Content and a shared navigation structure with a legacy Struts 1.1 application.

The Enterprise Portal Architecture for the customer is to migrate all legacy functionality from WebLogic Portal to WebCenter Portal over the next year in staged releases that will also include the introduction of new features and functionality.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

How Not to Succeed with Agile

Originally published at developer.com

Why Agile Projects Fail

The purposely provocative title is meant to capture the attention of people with two opposing mindsets. The first are the proponents of agile methods who will want to pick apart any arguments against their preferred approach. The second are those who oppose agile and are looking for justification for their concerns. The goal of this article is to reduce the opposition between these two schools of thought, and to provide some food for thought to those who are undecided.

The truth is development projects following agile methodologies have failed. Projects following waterfall and RUP have also failed. Just as the cliché that one should not judge a book by its cover is often true (I missed out on the intellectual pleasure of Robert Heinlein’s Friday for many years due to that primitive prejudice), one should also avoid drawing a conclusion about a topic based on the title alone (The Art of War is more about how not to fight).

We will not go into deep detail of agile methodologies today. There are too many to cover adequately in a single reading, and the specifics of these methodologies do not have a direct impact on why agile projects succeed or fail. Agile is as much a way of thinking about projects as it is a set of tools that can be applied to projects, and the aspects that make up successful thinking can apply to agile and non-agile methodologies.

The Project

Both definitions of agile in one online dictionary include the word “quick”. The word agile evokes thoughts of athletic prowess and graceful, fast feline predators. It is not surprising the people who chose to adopt agile methodologies for the first time do so when facing a tight deadline. Perhaps choosing a methodology by its name will be the new cliché.

People who have followed agile methodologies for multiple projects will tell you that they provide quick results, and can show you artifacts that prove it.What they may have forgotten (or simply neglected to mention) is that successful agile projects have a few advantages to be successful with agile methodologies.

For an IT shops’ first agile project to be successful, the project must be the right project for that shop to adopt agile techniques. The project should be small, in both time frame and team size. The managers of the project must be willing to either adopt agile deliverables as they are, or adapt their method of measurement. Most importantly, the project team either needs to have enough members already experienced in agile or be given the latitude to get through the learning curve.

Agile methodologies rely on continuous and open communication between all levels of stake holders.For traditional waterfall groups, it is unheard of for a developer to contact a business sponsor or client to find out what is meant by a particular requirement description. In an agile project, the ability to do so can mean the difference between success and failure.

Agile methodologies can be used successfully to deliver a large project, but not by a team that is learning to use these techniques for the first time. Large projects often include many team members, at multiple locations. An IT shop can grow a small agile team to a large, distributed team over a few projects, but it is likely to be disappointing to try to assemble a large team from scratch, even if all of the members are experience in agile. Like many adjectives, agile has different meanings to different people in different contexts, and two highly experience agile project members could have entirely opposing approaches. Large agile teams must be grown organically rather than assembled randomly. These caveats to applying agile projects are most likely the root of the general consensus that agile is not appropriate for large projects. Perhaps this caution is a good idea when introducing agile methodologies to a team for the first time.

In describing the ingredients of projects that can succeed with agile above, it is important to note that the different modal operators of probability used are chosen specifically, rather than simply to provide variety in wording. Project managers new to agile must be willing to adapt. Teams new to the agile approach as a team needs to either have enough members with agile experienced or be given the opportunity to stumble and learn. Communication across stake holders can impact results. There are exceptions, such as when those with roles that include being the translator of business requirements to technical requirements have good relations and communications with business and development and are highly competent in those communications.

This section began with a reference to a dictionary definition. In addition to the current English meaning of the word, most dictionaries include the history of a word, its etymology. At that same reference link, the etymology of agile is given as “from Latin agilis, from agere to drive, act”. So, apparently the meaning of agile in the English language evolved from its origins to include the notion of “quick”. Agile methodologies can evolve within an organization to be what we want them to be, though it is unlikely that they can be exactly what is desired the first time out.

The People

The introduction mentioned that there are opposing mindsets about agile. It also included a reference to The Art of War, and these are not coincidences. An agile project team with members of both camps has two strikes against it from the start. It only takes one more strike to be out.

Many proponents of agile tend to be over zealous in their belief of the superiority of their preferred agile methodologies. While their enthusiasm may win support from the undecided, it can also cause those opposed to agile methodologies to push back even harder. It can also frighten those who are participating in an agile project for the first time, as agile methodologies are very different from waterfall approaches and change causes anxiety for many. Where patient mentoring will frequently overcome FUD (Fear, Uncertainty and Doubt), ignoring the FUD of others generally results in greater FUD.

Most opponents of agile are against it because of FUD. Sometimes rightly so, as agile is not a panacea and is not the perfect methodology for every combination of projects and teams. Those most opposed to agile methodologies are non-developer team members. Experienced waterfall project managers and analysts are highly trained in creating complete definitions of deliverables and measuring progress towards those deliverables with specific tools. While agile is still a measurable methodology, the measurements are defined as the project progresses instead of up front. This can be too far outside of the comfort zone of some people. And people who are uncomfortable will often do whatever it takes to become comfortable again. For some, that is achieved by adapting and learning, i.e., becoming agile themselves. For others, the approach is to make every effort to bring things back to the way they are used to. These latter individuals most likely do not do it consciously, but they will sabotage an agile project. How? By doing exactly as they are trained.By trying to capture every minute detail for a story definition rather than outlining it and providing feedback during the daily testing. By trying to fit too many features into a single iteration. By thinking a milestone was missed because an iteration did not deliver all of the features planned.By not delivering requirements that are completely defined because the due date for the requirement has not yet arrived on the calendar. By running an agile project in a waterfall mentality. In short, by not being part of the agile process by being agile themselves.

To repeat, those who oppose agile processes and sabotage agile projects most likely do not do it consciously. If everyone isn’t onboard with the agile process at the beginning of the project, it can still succeed. If everyone isn’t onboard with agile by the end of the project, that project will most likely have been a failure. If everyone pulls their own weight, but pull in a different direction, something is bound to come apart.

Those who have delivered successful agile projects have good reason to be confident in their approach. Successful agile projects result in solid software, often delivered either under-budget or with more features than originally projected. The iterative approach of providing rapid, demonstrable deliverables quickly builds enthusiasm for both the clients and the developers. Those who began the agile project with FUD and then learned to adapt and adopt new processes are those who become the biggest supporters of agile. It is important that they remember that they had their doubts at first if they are to convince others who suffer FUD that they, too, can become successful agile project members.

Hopefully you realize that this section has many redundancies to the first section. Project deliverables may be software and documentation, but it is people who deliver them.

The Process

Waterfall projects are successful when there is a rigid adherence to process. To some, that rigid adherence may be the only ingredient they are aware of. They may not acknowledge that the project also required the experience and knowledge to create an accurate project plan with achievable milestones.

In agile projects, rigid adherence to process is a key ingredient to failure. Agile is made up of many different processes that can be adopted, combined, added, dropped and modified as needed. First-time agile projects are prone to the mistake of choosing their processes and sticking to them even when they don’t work.

A perfect example is Test Driven Development (TDD). In TDD, automated test procedures are written first, then the development is done and tested continuously. This is a great approach to ensure functional quality. It works perfect for the Control layer of an MVC architecture, for example. However, for the view layer of a web project, it can result in deliverable falling out of synch as it takes much longer to write tests for web pages than it does to write the page itself.

Daily stand up meetings are an excellent communication tool. Run properly, they provide an excellent benefit. However, if one or more of the unconscious saboteurs join in (or, as is often the case, lead) these meetings, they can become a key factor in iteration failure. Choosing to make the stand up every other day or weekly while leveraging instant messaging can put an iteration back on track if the daily stand up is detracting from productivity.

Another fallacy held by extreme supporters and detractors of agile is that it cannot be combined with waterfall. Nothing could be further from the truth. An agile development team that begins prototyping based on the initial, high-level requirements of a waterfall project will be able to demonstrate their achievements earlier in the development stage and get immediate feedback from clients. Because another fallacy about waterfall is that all of the requirements are defined before development begins. Change orders are part of the waterfall process, and it is very rare they aren’t used. The earlier the change order occurs, the better the chances of success are, and agile development processes within a waterfall project facilitate these early changes in direction that lead to improved deliverables.

Agile processes work if implemented in an agile manner and followed by people willing to be agile.

Conclusion

Why do agile projects fail? When agile methodologies are not used in a project labeled as agile. Whether you consider the root of the word agile as meaning to take action and get immediate results, or the modern meaning of quick deliverables, simply using the word agile is not enough to take action or be quick about it. Nor is simply taking quick action truly agile, unless it is done with purpose and support. Many of the most successful agile projects I have participated in were not labeled agile; we simply followed agile methodologies as a development team and met waterfall deliverables, usually ahead of schedule.

Agile projects do succeed, and agile methodologies are designed for success. Like any tool, if it is not used properly it will not accomplish what it is designed for, and can cause actual damage. If you have a fly to swat, a fly swatter works well… unless it is tied to a Buick.

Additional Reference Links

The Wikipedia entry on Agile at http://en.wikipedia.org/wiki/Agile_software_development contains many disclaimers, making it more honest than most references found during the writing of this article.

A short list of SDLC approaches can be found at http://testdog.com/knowhow/Sorting%20Out%20SDLC.html. While there are many, many other sites about various SDLC approaches, this one seemed to have the least agenda and the most variety.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson