I used to think the only reason why Scrum is so popular today is coz of their “over-the-counter” certification programs. After thinking about it, I started to question myself, why others cannot see that the whole certification thing is so flawed? My best guess was, if I can see, I’m sure a lot of other people can also see through it. Then why is it so popular?

I talked to close to 50 companies that claim they are practicing Agile (read as Scrum). I tried to understand what was the driving factor for them to choose scrum. Some obvious points came across:

• Its easy to get Management buy-in coz Scrum is big on Project Management aspect
• Scrum is very simple, so we can start doing it without a huge change to our existing organization structure and culture
• Its easy to find training (certification programs)

Most people said it was a good starting point. After doing Scrum for a while, we realized Scrum alone will not help much, we need some XP practices for sure.

The I asked them, why did they not choose to start with XP? What was the biggest concern?

Surprisingly most people said that, we have heard that in XP you don’t need Managers. Most people who do XP, claim that middle-management becomes a big road-block for them. In most organization they cannot get rid of middle-management nor can they find something more interesting/valuable for them to do. So XP becomes a big No No for them. But with Scrum, what lots of organizations do is, they send their managers to Certified Scrum Master classes and make them Scrum Master. Some become Scrum Masters on teams, some become Scrum Masters for their Scrum of Scrums. This way they get middle-management on board.

So looks like Scrum has found a way to address the role of middle-management. Hence fill the gap. I’m not too sure if I agree with this, but its a fact of what I see out there happening.

Disclaimer: Personally I don’t care what people do, XP, Scrum, Crystal, DSDM, RUP, Spiral, V&V, etc. As far as it really helps them and constantly improves them its great. But if they are just doing/following some process for the sake of the label, it really bother me.

Often people ask me what are the tangible results of agile software development?

Following is my elevator pitch :

In my experience Agile leads to :

• Happier and more satisfied Customers and End-Users due to faster turn around time and greater visibility/involvement
• Happier development team due to heavy focus on people and team.
• Greater emphasis on quality (and testing) by building it into the process rather than having it as an after thought
• Self organization and empowerment of team leads to more responsible team and greater innovation
• Very effective at handling changes in requirements, technology, people etc
• With the large number of feedback cycles at various levels, it is very effective at adapting the project progress and hence addressing project risks

Obviously there are many more points, but these come to my mind immediately.

At XP 2008, during a panel on “There’s no such thing as Best Practice”, Joseph Pelrine talked about new emerging Agile methodologies called “XPLite” and “ScrumBud”. (heavily influenced by MillerLite and BudWeiser beer) Accordingly to him, there are lot of teams who claim they are Agile because they follow one of the most important Agile principles i.e. “Inspect and Adapt”. They read the book (inspect) and then adapt it by taking and applying what they think is most convenient to do.

Today Distributed Development is unavoidable and is the need of the hour.

My belief is that if you have the “right” set of people, motivated and happy on your team, it will be a huge contribution to the success of the project. But the problem is most often “right” people are distributed all over the world and not located in the same city. A lot of organizations have tried to move people to the same city so that they can be collocated. While this might work to some extent, this model cannot scale and is not sustainable. All of us would like to settle down in the location of our choice. We don’t want our work to dictate where we live and what lifestyle we choose. For example, I like to live in India close to friends and family. So far I have traveled quite a bit, but once my daughter starts going to school it would be very difficult for me to relocate to another city.

Also if you consider a lot of leading companies they are loosing a huge amount of people, because most of those employees don’t want to travel/commute any more and want to settle down in one place. While tele-porting is a few centuries away, employees have to live with stressful commute everyday. Not only this is a huge wastage of time, it also affects your productivity and makes your family life very stressful. Currently most of the successful companies have offices in multiple cities. One of the main drivers for having multiple office in different cities is to attract local talent. If your company does not plan to have a office in location X, guess what, your competitors will open an office and you’ll loose some of the smartest folks you could have other wise hired. And when you have multiple office, guess what, people in different offices have to work together. This is another reason why distributed development is becoming more and more important.

In Agile we talk about having development and business teams together. It is very critical to have constant business involvement. But what do you do when your Business is distributed? Because of the global economy most businesses have global presence.  Also each region might have slightly different business rules or market. Hence it is very important to have development teams close to local businesses and distributed. This again leads to distributed development.

Another point to be aware of is, if we consider what industrialization has done by building power centers in industrial cities leaving the rest of the country really backward. We don’t really want IT to do the same thing to the country’s economy.

If you consider these practical issue, collocated team model turns out to be unrealistic in the long run. The chances are you will have to compromise on the quality of people or have the team members travel a lot if you want all of them to be collocated. The alternative is to build a model where the team members can be distributed across space and time.

Unfortunately so far, we’ve not had great results with distributed development. A lot of people have published their experiences trying to effectively work in distributed teams. For some it has worked or some it has not. But for most of them its not as effective as collocated teams.

What has worked best of me so far is to start off a team by having them collocated for a couple of week to a month. Then created cross-functional self sufficient teams in each distributed location. Once this model seems to work, then try to fully distribute your team with team members spread all the world. This turns out to be a long and painful process. But this is the best I know of today. Check out my presentation on Distributed Agile which I gave at the IV Agile Gathering in Ukraine.

Next year I plan to kick off a few experimental distributed projects to try out different techniques to get Distributed Development work as effectively or may be even better than collocated teams.

At the last Agile Coach Camp I gave a lightening talk on “Process OVER People”. In this talk I requested all the coaches present at the conference to really think about their advice to companies. Are we trying to put a process boundary and become Process police? Is this coaching? A lot of coaches I meet, recommend “First do it by the Book, then deviate”. What does “First Do it by the Book” mean? I appreciate the book and I think there is a wealth of knowledge out there. But we should remember the book was written with some context in mind and at some point in time. One cannot blindly take what is in the book and try to apply it. That really feels like “Process OVER People” to me. You need to take your team into account. You need to consider what you are trying to build and most importantly you need to prioritize what needs to be fixed on your team or in your organization before trying to go and “Do it by the Book”.

Over the last couple of years I really feel Agile is gone into a mass-production mode and we’ve stopped innovating. Every company wants to be Agile. This has lead to a huge demand for Agile Coaches. And because of this you find all kind of people claiming to be Agile coaches. What surprises me is a lot of these folks have themselves never really worked on an Agile project. Forget Agile project, a lot them don’t really have a successful project delivery story to share. How can one preach something and do something else (or do nothing)? My belief is that one needs to lead by example and not my blabbering crap.

I asked the participants at the conference to tell me what new tricks, techniques, tips, practices, etc they have learnt in the last one or two. Very few people (may be 2 out of 50) had something to share. Are we getting so busy mass producing what onces worked that we have forgot to go back into the trenches and try new things? My fear is that if we continue this way, Agile will soon be the next CMM (at least the bad implementations of it, which is most common).

If you are interested in some background about where I’m coming from, you can read my position paper for the Agile Coach Camp.

Why are companies so afraid to Fail?

When I meet folks from companies, most of them want to implement Agile, but they want ready made solutions from experienced folks. Basically what they are looking for is a Dummy’s Guide to Agile Transition. They want a completed tested and proven approach (Best Practices) to adopt Agile. They want to make sure there is no room for failure.

Well I don’t really understand how can one learn new techniques/approaches without failing a couple of times? Isn’t failure an implicit part of learning? To really learn something, you need to understand its boundaries and test the waters yourself. Babies don’t learn to walk without falling/hurting themselves. IMHO, if you want a babysitter (Coach), all your life, you will never be able to appreciate and learn new concepts. I’m afraid you will learn how to be waterfall in a different way.

You could get a coach to help you avoid big failure and give a helping hand after a failure, but its unrealistic to expect that a coach will help you successfully transition without any failure. Hard learnt lessons stick around longer.

We have just finished with the Penn State University Project. This project was a way for me to evaluate the reverse sourcing phenomena and to make a difference to our education system by playing a demanding customer’s role for a University project. Overall I had a great experience doing this and the students really enjoyed working on this project. (I owe a separate blog to talk about the overall project, now that it is over). In this blog, I want to highlight something that I always felt about user stories but was not clearly able to articulate.

The Mystery of Shrinking Stories:

Over the last 5+ years of my experience with User Stories, I’ve seen people constantly trying to reduce the scope (size) of the user stories. (Jeff Patton has a really interesting blog about the same). Seeing this trend I’ve been frustrated a lot of time when my customer or team members don’t understand what is happening.

The problem was the stories were too granular that it was easy to miss the big picture.

Functional Areas :

To avoid this problem we started grouping the stories into functional areas. The Product Owner would identify the highest priority functional area and we used that functional area to identify our iteration/sprint stories. The developer would come up with a list of stories and tasks that fit under that functional area. (Mostly this would be done before the planning meeting). In the planning meeting we would have a collaborative session with the Product Owner to decide what stories need to go into the iteration/scrum based on the estimates. But we still had a big estimation session and then we would identify a list of stories. This clearly simplified our planning meeting, but

• The team members still did not have 100% clarity during the iteration/sprint on why certain stories made it and some did not.
• Both sides felt like there was a compromise in terms of the identifyed stories. The Product Owner might want some story but the developers would say its not possible because the estimate is high on it. The Developers might want to work on some story because it makes sense from a technical point of view, but the Product Owner did not prioritize that story.
• Also another problem we ran into was there were a lot of times when we wanted to pick stories from different functional areas. But in this approach we could pick up stories only from one functional area.

Using Themes instead of Functional Areas

We used different approaches to identify Themes.

1. First we started off with identifying a list of high priority stories and then tried to put a theme around it. This clearly did not work as well.
2. Some times we tried to identify a list of high priority functional stories and use a theme for the technical stories. This worked well when the technical debt was really high. For Example, Refactoring is a theme for this iteration/sprint.
3. Finally we asked the customer to identify a theme and the developers would then identify stories based on the theme. This was very similar to the functional areas except that now we could identify stories from different areas.

Again we had similar issues with previous approach.

Each Iteration/Sprint has a clear Goal

Finally we tried having a goal per iteration/sprint. The Product Owner would communicate the goal to the team; the theme would discuss about the goal with the PO and get their clarifications. After which the developers would come up with a list of stories that will help them achieve that goal. Sometimes they might only be able to achieve part of the goal, and then the goal will be updated accordingly.

This gave the business folks to clearly communicate what they wanted to see at the end of the iteration and also that fits into our release goal (each release had a clear goal, but before this we did not have a goal per iteration/sprint). For the developers this meant they could thin slice the stories accordingly to fit into the iterations. Suddenly developers moved from an estimation mindset to a budgeting mindset. Also from the PO’s perspective the story is really the goal, which is fairly high-level and the developers used user stories as tasks to keep track of progress and to split work amongst them. I was never a fan of tasks under the stories and this helped us get rid of them. Technically user stories (not really give by the user, written by the development team from user’s perspective) became our tasks and the goal became our user stories. This way we were able to make our stories quite large in scope (really, its one story per iteration/sprint). There is a similar concept in Scrum. They talk about Goals for a sprint, but there are multiple goals per sprint. In our case we had only one goal per sprint and we had stories under that.

What we did on the Penn State Project:

We started off with me (Customer or Product Owner) explaining them the overall vision of the project. So that, all of us has the same mental model. We used a mindmap for this. Next, I started of setting a goal for the iteration and also giving them a list of high priority stories (since they were new to the whole concept of stories). After a few iterations, I just gave them goals, asked them to email me the stories the next day and then in our bi-weekly demos, they would demonstrate each story and I would get to rate if they fulfilled the goal or not.

Some sample Goals are:

• Have at least one conference up and running
• Build versioning into the system ( This meant they had to add version each page, each artifact under the page, version templates, version the whole conference, there could be multiple conference under one site, so version all of them)
• Add template support (This means layout templates, Style sheet, include template, etc)

This has been my experience of moving away from Granular User Stories to Goals. I would love to hear your experience.

I just got back from Chandigarh after participating in the Agile Chandigarh 2008 workshop.

Vikram Dhiman did a great job of organizing the workshop. I really appreciate his initiative.

We had over 35 participants from 11 different companies. I was surprised by the level of Agile knowledge the participants had.
I ran the Waterfall to Agile demo followed by Agile Overview presentation. After this, Vikram gave a great presentation on Agile Myths. The last presentation of the day was from Rajesh. Rajesh spoke about the role of a manager. Rajesh was just introduced to Agile by Vikram a couple of weeks back. Rajesh has been doing Management consulting for the last 15+ year and he was happy to see that Agile is embracing what he calls “Common Sense” management.

Vikram will upload all the slides soon.

Thanks to all the participants and organizers. I wish more people take up initiatives like this, so that we can spread the agile awareness faster and move up the value chain.

Just noticed that the Agile Alliance Group that I has created in LinkedIn is growing massively. We have 1403 members so far. If you believe in Agile, join this group and show your support.

To join this group on LinkedIn click the following link: http://www.linkedin.com/e/gis/37631/0FF74232FB92 (You’ll need a LinkedIn Id. If you don’t have one you can always create one for free)

I hope joining this group will activate the consciousness of belonging to the Agile community as a network.

Why do companies spend enormous efforts in vain searching for “Best Practices”? Development Best Practices, Deployment Best Practices, Best Design Practices, Language Best Practices and so on….

I really feel uncomfortable when people talk about Best Practices. A lot of times we get stuck up in coming up with “best practices” as if they were final or ultimate. The whole point about Agile is, software development approaches are not static or final, we keep evolving and emerging. If something by nature is dynamic, evolving and emerging how can we say this is best? IMHO “best practices” exist only for a given context at a point in time (seconds). As soon as the time is over or the context has changed its no longer best.

Personally I prefer calling it “Better Practices”. For Ex: This practice in this context is better than the one we were using before this and hopefully tomorrow there will be a slight variation or a different practice which would be better than the one we are using today. There is no best, its only better.