How has Agile evolved over the last 12 years?
This is an introductory presentation on the essence of Being Agile vs. Following Agile. And why being Agile is important? I’ve also tried to show an evolution of Agile methods over the last 11 years and the future of Agile. Also take a sneak preview into what challenges an organizations may face when trying to be agile?
You’ve heard about limiting WIP (Work-In-Progress) but how good are you at limiting red time? Red time is when you have compilation errors and/or failing tests. A growing group of practitioners have learned how to effectively reduce red time while test-driving and refactoring code. To understand how to limit red time, it helps to visualize it.
In this talk, I demonstrated various strategies to limit your time in Red. We also analyzed a live programming sessions using graphs that clearly visualize red time. Participants learned what development processes help or hurt our ability to limit red time and gained an appreciation for the visual cues that can help make you a better developers and fellow member of the Limited Red Society.
Slides from the Presentation:
The first part covers the period from 2004 to 2006 when Sidd was working with a startup based out of Singapore. He explains how we moved from doing ad-hoc development to adopting Scrum. Adopting Scrum was a big improvement over our previous ad-hoc approach but Scrum also led them to make some classic mistakes (from a lean point of view).
The second part covers the period from 2007 to 2009 when Sidd started his own company in India. The company was started with Scrum right from the beginning. He explains how we evolved from vanilla Scrum to Lean and Kanban.
Developing nations, which are trying to establish themselves as a IT Destination have to fight against the giants in the out-sourcing world. For many years companies used CMM as a way to get some attention. But CMM is
- heavy weight,
- expensive to get assessed
- does not really fit small companies’ (20-50 people’s) needs.
- Also in the last few years CMM has got a pretty bad reputation and hence its not necessarily a great marketing advantage.
Considering all this, what do companies do? What is the new shinny thing they should all run after. ..ta..da… Agile & Lean.
- More suitable for their needs,
- Some Customers are asking for it
- Is also trendy and gives the marketing advantage over the big outsourcing companies.
Following is a list of random topics (most of them are Agile/XP practices) about this project:
- Project Inception: We started off with a couple of meetings with folks from Freeset to understand their needs. David quickly created an initial vision document with User Personas and their use cases (about 2 page long on Google Docs). Naomi and John from Freeset, quickly created some screen mock-ups in Photoshop to show user interaction. I don’t think we spent more than a week on all of this. This helped us get started.
- Technology Choice: When we started we had to decide what platform are we going to use to build the site. We had to choose between customer site using Rails v/s using CMS. I think David was leaning towards RoR. I talked to folks at Directi (Sandeep, Jinesh, Latesh, etc) and we thought instead of building a custom website from scratch, we should use a CMS. After a bit of research, we settled on CMS Made Simple, for the following reasons
- We needed different templates for different pages on the site.
- PHP: Easiest to set up a PHP site with MySQL on any Shared Host Service Provider
- Planning: We started off with an hour long, bi-weekly planning meetings (conf calls on Skype) on every Saturday morning (India time). We had a massively distributed team. John was in New Zealand. David and Deborah (from BestBuy) were in US. Kerry was in UK for a short while. Naomi, Kelsea and other were in Kolkatta and I was based out of Mumbai. Because of the time zone difference and because we’re all working on this part time, the whole bi-weekly planning meeting felt awkward and heavy weight. So after about 3 such meetings we abandoned it. We created a spreadsheet on Google Docs, added all the items that had high priority and started signing up for tasks. Whenever anyone updated an item on the sheet, everyone would be notified about the change.
- User Stories: We started off with User Persona and Stories, but soon we just fell back to simple tasks on a shared spreadsheet. We had quite a few user related tasks, but just one liner in the spread sheet was more than sufficient. We used this spreadsheet as a sudo-backlog. (by no means we had the rigor to try and build a proper backlog).
- Short Releases: We (were) only working on production environment. Every change made by a developer was immediately live. Only recently we created a development environment (replica of production), on which we do all our development. (I asked John from Freeset, if this change helped him, he had mixed feelings. Recently he did a large website restructuring (added some new section and moved some pages around), and he found the development environment useful for that. But for other things, when he wants to make some small changes, he finds it an over kill to make changes to dev and then sync it up with production. There are also things like news, which makes sense to do on the production server. Now he has to do in both places). So I’m thinking may be, we move back to just production environment and then create a prod on demand if we are plan to make big changes.
- Testing: Original we had plans of at least recording or scripting some Selenium tests to make sure the site is behaving the way we expected it to. This kind of took a back seat and never really became an issue. Recently we had a slight set back when we moved a whole bunch of pages around and their link from other parts of the site were broken. Other than that, so far, its just been fine.
- Usability: We still have lots of usability and optimization issues on our site. Since we don’t have an expert with us and we can’t afford one, we are doing the best we can with what we have on hand. We are hoping we’ll find a volunteer some day soon to help us on this front.
- Versioning: We explored various options for versioning, but as of today we don’t have any repository under which we version our site (content and code). This is a drawback of using an online CMS. Having said that so far (been over a year), we did not really find the need for versioning. As of now we have 4 people working on this site and it just seems to work fine. Reminds me of YAGNI. (May be in future when we have more collaborators, we might need this).
- Continuous Integration: With out Versioning and Testing, CI is out of question.
- Automated Deployment: Until recently we only had one server (production) so there was no need for deployment. Since now we have a dev and a prod environment, Devdas and I quickly hacked a simple shell scrip (with mysqldump & rsync) that does automated deployment. It can’t get simpler than this.
- Hosting: We talked about hosting the site on its own slice v/s using an existing shared host account. We could always move the site to another location when our existing, cheap hosting option will not suit our needs. So as of today, I’m hosting the site under one of my shared host account.
- Rich Media Content: We questioned serving & hosting rich media content like videos from our site or using YouTube to host them. We went with YouTube for the following reasons
- We wanted to redirect any possible traffic to other sites which are more tuned to catering high bandwidth content
- We wanted to use YouTube’s existing customer base to attract traffic to our site
- Since we knew we’ll be moving to another hosting service, we did not want to keep all those videos on the server which then will have to be moved to the new server
- Customer Feedback: So far we have received great feedback from users of this site. We’ve also seen a huge growth in traffic to our site. Currently hovering around 1500 hits per day. Other than getting feedback from users. We also look at Google Analytics to see how users are responding to changes we’ve made and so on.
- We don’t really have/need a System Metaphor and we are not paying as much attention to refactoring. We have some light conventions but we don’t really have any coding standards. Nor do we have the luxury to pair program.
- Distributed/Virtual Team: Since all of us are distributed and traveling, we don’t really have the concept of site. Forget on-site customer or product owner.
- Since all of this is voluntary work, Sustainable pace takes a very different meaning. Sometimes what we do is not sustainable, but that’s the need of the hour. However all of us really like and want to work on this project. We have a sense of ownership. (collective ownership)
- We’ve never really sat down and done a retrospective. May be once in a while we ask a couple of questions regarding how something were going.
Overall, I’ve been extremely happy with the choices we’ve made. I’m not suggesting every project should be run this way. I’m trying to highlight an example of what being agile really means.
While everyone agrees with the value of eating your dog food, some people claim that this principle cannot be applied to all software industries.
Let’s take the Medical Health Care Industry. Who should build software for Doctors and Nurses to be used in the hospital? Its very unlikely that Doctors will start building software in the side. How do apply this principle here?
What we have today is a bunch of people trying to build software for the hospitals (most of them have not clue on how a hospital operates, those who know a little become Subject Matter Experts and take charge). Similarly there are lot of other industries.
You ask their users how they like the software and you would know. Its not that the development team did not do a good job of building the features right or the business did not do a good job of articulating what they want well. Its just that this model is setup for failure.
- The Agile community realized that, they need to bring the users in and collaborate with them much more.
- The Scrum community identifies one person or a group, call them Product Owner. They are part of the planning meeting, daily scrum and even the retrospectives & demos. Some (0.1%) of teams are able to get actual users during their demo. Are they confused about the PO being their User?
- The XP community demands an onsite customer who can guide the team not just during planning, but also during execution. Again the same confusion exists. But the situation is slightly better.
- Having said that, I really appreciate XP for pushing the knob on automated testing. Automated Testing (esp. Developer testing) is a great way to eat your own dog food. Remember how useful your API tests have proved to be. Tests are clients to your code and they consume your code by acting as client.
- The Design (UX) Community are lot more User focused and tend to spend more time with the actual Users, but that’s very sporadic
- The Lean community have realized that they need to have the development team sit with the business in their work area. They have realized that there are a lot of important lessons to learn from the context of the work place.
Personally I think we need to go way beyond this. If you look at some organizations (esp. Web 2.0 companies and Open Source Projects) they are their own Users. We can certainly learn something from them.
How can we do this? Here are some ideas:
- At least to start with, have the team members take a formal education in the domain they are building the software. Do some case studies and then, spend quality time with the Users (actual Users). Not just interviewing them, but actually working with them (at least shadowing them or being their apprentice).
- Educate the Users more about Software development process and have them work with the team for at least a week or two to under it.
- May be hire people who have actually worked in the field. (You want to make sure their knowledge is up-to-date and they actually know the business really well). Also very important to maintain a good ratio. 1 member for a 10 people team is scary.
- Build tools that can help the actual end users build/configure their software. As developers we build tools which we use on our own projects. Same tools (which were driven by eating their own dog food) can now be used by others to build their software. For years, creating a web presence for a company was a specialist’s job. Today with Google Apps and others, anyone can set up a website, add a bunch of forms, set up email accounts and all that Jazz. The line between a specialist’ role and a business user is blurring. Coz we have the tools to help. Esp. tools built by people for their personal use.
- Again all of this can get you one step closer. But nothing like eating your own dog food.
- Focus on Rapid Delivery: No one likes to wait for food. No one likes to get cold, stale food. Food is best when its served fresh and hot. The exact same thinking applies to software. If I go to a restaurant and order for starters, main course and deserts, its stupid to expect that I’ll oder everything one-shot and accept everything delivered together. Most probably I’m going to order the starters. Once you get the starters, I’ll see it’s quantity, also taste them and then decide on the main course. And so on. From the restaurant’s point of view,
- They don’t get nor expect all the requirements upfront.
- They believe and embrace iterative and incremental delivery model.
- They deliver food as and when its ready. (focus on throughput). In fact restaurants in India, serve you really fast and they want you to eat and leave as quickly as possible, so that they can server more customers. They really focus on throughput and flow.
- They keep their customers busy (hooked in)
- They don’t want to keep the food waiting to be served (inventory)
- Innovation: Profits and Competition are very high and hence restaurants are very innovation driven. They know only good food is not sufficient to keep customers loyal. They constantly do the following to attract repeat orders:
- Chef’s recommendation and today’s special deals
- Free Home Delivery
- Improve interiors and ambience
- Come up with appealing offers and packages
- Evolve their menus by adding new items to their menus and food offerings. Constantly try to improve it based on most frequently ordered items
- Heavy focus on service and customer satisfaction
- Try to build a personal rapport with each of their customers. Make them feel special when they come.
- Constantly look at eliminating waste.
- If plates, spoons, knife, forks and tissues are frequently required, they store them very close to each table. So that they can avoid their movement and hence save time.
- Try to make their order taking, processing and deliver process more efficient and less error prone (mistake proofing).
- They divide their responsibilities into various roles like Order taking, delivering food and cleaning up the table. (Same thing might not work well in software because intrinsic knowledge is much higher)
- The Chefs inside the kitchen learn to keep their work-area clean so that they don’t get caught up in the mess, trying to find things they need.
- Chefs also do a lot of interesting mistake proofing to avoid confusion between ingredients
- Restaurants watch food consumption trends and prepare (plan) their food ingredients based on those patterns. For Ex: they know on weekends, they’ll have huge demand, they plan accordingly. (avoid inventory)
We have also seen how a small, really successful restaurant starts scaling by opening franchisees and soon its brand is completely destroyed. Be aware of the scaling black-holes.
Kanban seems to be a good starting point to adopt Lean thinking on a team. Of course Kanban alone is not sufficient to become lean. You need to Trust & Respect team members and you need concepts like Single Minute setup, Mistake-proofing, Zero Inspection, Kaizen, etc. Kanban to me is a great manifestation of Queuing Theory and to some extent Theory of Constraints.
Jeff Patton wrote a great intro to Kanban.
Of late I’m hearing a lot about Kanban and its application to software development. IMHO Kanban is nothing new. If you look at a lot of “traditional” maintenance and support teams they’ve been using Kanban for ages.
Back in 2004 when I was leading an offshore maintenance project without knowing anything about Lean or kanban, we really evolved to using a pull-system (Kanban) on our team. That was the only logically way we could work. Of course we started off with Iterations and Releases and so on. But we quickly implemented a simplified Kanban on our team.