Is testing really a safety net?

I was having a discussion with Steve Wilhelm, one of the most experienced QA from ThoughtWorks, about testing as a safety net. Steve bought up a very nice point which set me thinking. He expressed that, testing should be embedded into the development process and should not considered as a safety net. The point is to bring testing and development closer together. The closer they are the better is the feedback cycle.

The analogy of safety net has two aspects to it:
1. The net is already built and is safe to fall back on. I yet have to see a project which had all the testing ready before the development started. They usually go hand in hand. In fact most of the projects are struggling to get to this point from the old waterfall fashion
2. The net is an independent reusable entity which can be applied any where. Alas, the whole testing strategy has to be crafted for each project and is very tightly coupled with development. “Throw it over the wall” philosophy does not seem to work.

Next time you think of testing, reconsider this analogy.

Greatly influenced by the CIT conference, with the help of folks from the Agile Philly user group, I‘m planning to organize a similar conference on Simple Design and Testing in the greater Philadelphia area. I‘m planning to call it SDTConf 2006.

There is an update on the conference, please check Updated Blog Entry or Conference Website for details.

The Agile Alliance is supporting and sponsoring this conference.

So far what I have planned:
1. Format – Open spaces. Very similar to what we had at CITCon. I would also like to see if we can have some hands of sessions.
2. Dates: Oct 27th – 29th 2006.
3. Venue: West Chester University
4. Theme: Simple Design and Testing. There has been a lot of discussion about simple design, but I have not really seen that happen on large projects. I was wondering if we can learn from each other‘s experience of how to tame this beast. I also want to know whether people recognize the importance of testing and how it helps in simple design.
5. Conference fee = $0 . I want it to be a free conference
6. Delegates: Max 70 people.

As far as I can think, following are the things I would need to make this conference happen:
1. Good Participants: Very important.
2. Venue: University. [At least 2 halls + one lab]
3. Food and drinks
4. Conference website – http://sdtconf.com
5. T-Shirts and other goodies

Based on my rough calculation its looks like the budget for this conference would be around $4k.
1. Food [$3k]
2. T-Shirts and other goodies [$1000]

Since this is based on Open spaces, we need not worry about flying speakers down or any of those.

Based on the above calculations, we would need at least 2 sponsors for this conference, each contributing $2k. Agile Alliance has offered to sponsor $500.

Please drop me a note if you are interested.

I‘m not an expert in setting up and running user groups. But these are my experiences starting a few agile user groups in India and Philly.

Following are important ingredients for any user group:

1. Vision or Goal
2. Facilitator
3. Body of knowledge
4. Discuss topics or content
5. Individuals
6. Venue
7. Offline discussion forum

Vision: The primary vision/purpose/goal of any user group is to provide a platform for the members to share and learn from each other’s experience. Its a peer-to-peer learning and exploration platform. I like to call this SCube = Speak, Share and Seek.

Facilitator: Running a user group needs one or more facilitators who understand this vision/goal backed with a lot of passion, motivation and energy. Their primary purpose is to build a self organized group without trying to be gatekeepers. The facilitator need not be an expert on the topic. What I have seen happen in most of the cases is:
“A person is interested to learn more about a subject and is looking for ways to learn from other people’s experience. S/he is unable to find an existing platform to exchange ideas with other people. So they go the extra mile to set up a platform for people to meet”.

Starting a group is relatively easy. The real fun is in sustaining the group. As the group starts growing, one has to make sure the right topics are been discussed. They also need to keep people’s egos under check as the group grows bigger.

Body of knowledge: Every user group needs some source of information for the following

• to drive the topics
• to get external speakers and
• to help with the logistics.

The internet can be a very good source of this knowledge. It can also be very helpful to have a body that supports such initiatives. For examples in India, ASCI can help you and support you if you want to set up Agile User Group in your city. Right from financial support to logistics and contacts, all of this information/knowledge can be provided by these bodies. They can also help the group to be focused.Discussion topics or content: This is the real meat for which people would be interested in the user group. The group needs to have a good mix of the following:

• Presentations: Good for introductory sessions and for presenting experience reports. Helps to bring people on the same page in terms of terminologies, vocabulary etc.
• Discussions: There are different formats of discussion which can be helpful:
  • Bird of Feather
  • Open spaces
  • Fishbowl
• Activities/Games: This is one of the best ways for people to experience the topic. Folks have found innovative ways of explaining both technical and methodology related topics. For Ex. Agile related topics XP Game or Waterfall to Agile Demo
• Hands on sessions or workshops: While the participants can benefit the most out of these sessions, it‘s also a lot of work to set this up. In the agile space the most common workshops are:
  • TDD workshop
  • Refactoring fest
  • Coding dojo
  • Coding Kata
  • Testing workshop: Any one testing tool is selected. The participants are given with an application to test using this tool.

One should try to involve experience people to present/facilitate some of these discussions. In fact I have seen most innovative ideas come out when someone from the particpants come forward to facilitate the meeting.

Individuals: Having a small group of motivated individual willing to self organize. Most importantly they should be eager to learn and open to share knowledge. Having a diverse group of people from different companies/backgrounds can maximize the learning for everyone. When starting a new user group I have always found it to be helpful to not focus on numbers. A small set of motivated individuals is more important than a hall full of “I don’t care” individuals.

Venue: Any decent place convenient for majority of the group to meet. Ideally a conference room at someone‘s office or a classroom in some university or just any place can work. If you are using office space for meeting, be sure to rotate the venue between different similar companies to avoid one company‘s monopoly.

Offline discussion forum: A wiki and/or a user group on Yahoo or Google can provide the platform for offline discussion and collaboration. Any logistics related info should be radiated thru this forum. Great discussion happens on mailing list, but one needs to be aware of their limitations due to deterioration in communication bandwidth.

How to get started?
I believe in starting small. Trying to get a small, focused group of motivated individuals meet regularly and discuss interesting topic. Topics than can help them right away at work or else where. One should be able to find few people in their city or near by, who have experience on the topic. They can share their experience.

Once you have a small group in place, you should form a mailing list and start the initial discussion. Hopefully someone from this group should be able to get a venue for the meetings. Coffee shops, bars, office spaces, university class rooms are all good to begin with.

One should try and find people in the group who can volunteer to present/facilitate a topic/discussion. To start, it would be good to have a few nice topics on hand and then let the group decide what direction they want to take after that. Just discussions can get boring; organizing some activities can really help break the ice. In addition to these activities it would be good to find someone who can facilitate a workshop or some hands on sessions.

Asking people for suggestions about books, sites, articles, blogs, etc can help to get started.

Abstract: Introduce the participants to real world refactoring by taking an open source project and refactoring it. This session will help people understand
1. How to identify code smells?
2. How to eliminate the code smells by applying refactoring techniques explained in Martin Fowler‘s book?
3. How to refactor in absence of unit and functional tests [Working effectively with legacy code base]?
4. How to refactor to patterns?
5. Internals of the open source project chosen to refactor

Type of session: Hands on session.

Prerequisites:
1. Good Object Orient programming skills
2. Experience with Unit testing using the xUnit framework. Experience with mock objects can be helpful.
3. Pair programming. The participants should be willing to pair program. They will need to pair and do the real thing
4. The group decides any one open source project in the language of their choice.
5. Laptops or machines should be setup with a decent IDE and the source code. Each pair should have at least one functional laptop/machine. Make sure the code compiles and works on each machine.
6. Projector to which all the machines can be connected one after other.

Nice to have:
1. Network connectivity
2. A server with version control and continuous integration server setup.

If this approach is chosen, you need to make sure that each pair checks out the source code from this version control repository.

Selection criteria for the open source project:
1. Functionality of the project is pretty simple and straight forward to understand. Ex. Log4j, simple wiki, etc
2. The project is being used extensively on the team. Will help them understand it better.
3. Has an average sized code base. Too huge or too small code base will not be effective.
4. Should be easy to setup and test on any machine.

Execution:
1. Plan for at least five, 2 hours sessions.
2. In the first session,
2.a. First 60 mins: the coach explains/shows different code smells and briefly explains the associated refactoring techniques.
2.b. Next 30 mins: the coach gives a high level over view of the open source project‘s design
2.c. Last 30 mins, the team breaks into pairs and each pair picks up a module/package of the open source project. The rest of the time is spent in understanding the functionality of the module/package in individual pairs.
3. From the second sessions onwards:
3.a. The team splits into pairs, picks one or more classes from the module/package, understand what it does, writes unit tests around it. Once they have sufficient passing unit tests, they start identifying code smells and refactoring them. In some cases, sufficient unit tests might not be required to do simple refactoring like extract method, rename variable, etc. Sometimes internally restructuring the code, can help us understand it better. Hence you need to constantly wear the understanding, unit testing and refactoring hat.
3.b. At the end of 30 mins, the team stops refactoring. Sequentially every team connects to a central project and demonstrates their accomplishment to the rest of the group. The next 30 mins are spent discussing each team‘s refactorings.
3.c. The pairs swap partners so that each team gets a new member and one old member continues on the selected module/package. And they continue refactoring.
3.d. At the end of 30 mins we have another demo and then end of session.
3.e. If we have the version control and continuous server setup then people check in their source code and make sure the build passes.

These sessions can continue as long as the team is interested and they think they are learning something new. At the end of these sessions, the team would have made several improvements to the open source project. These improvements can then be submitted back to the open source community. Having a version control and continuous integration server can really simplify this.

I ran these sessions at ThoughtWorks and will be running them at the Agile Philly user group shortly.

Debates are not a great ways to communicate concrete points. Debates usually tend to get personal and emotional, leading to a destructive dialogue and indecisiveness. Luckily I have found fishbowls as an alternative to these debates where people can indulge in a more creative communication or a real dialogue.

A Fishbowl is a way of structuring a discussion in a large group of people. It is suitable for controversial topics where different participants have different views. A small group, the initiators, (the fish in the bowl), typically 4 or 5 people, sit on chairs around a round table placed at the center of the discussion room. The initiators explain the topic and start the discussion. The audience, the other participants, is seated around the initiators in a circle. Any members of the audience can walk up to the bowl and take a seat when they feel they have a contribution to make. When all the seats are occupied, usually the initiator who has been in the bowl the longest leaves making room for the new member, but this can vary. The FIFO (First In First Out) rule seems to work well to manage the participants at the bowl.

Alternatively, one seat is always left empty at the bowl. When all the seats get occupied, one of occupant has to leave. The rule to decide who leaves can also vary. FIFO or the person who feels they are contributing the least can leave.

The biggest advantage of this approach is transparency. The whole discussion is out in public. It gives a nice way to manage a large audience who would like to participate in the discussion. By letting a limited number of people talk, it helps to minimize the communication paths.

There are two challenges I have faced. Firstly, since the discussion is open and others are watching, some people might feel shy and not contribute even though they have some valid points. Other challenge is to avoid the most vocal/influential folks from hogging the seats.

There is another variation of the fishbowl format which can be found at http://www.co-intelligence.org/y2k_fishbowl.html
I have not tried this. But I prefer the one I have explained.

In the Agile India 2006 conference, we had a panel discussion on “Fixed bid agile projects are oxymoron”.

People in favor of fixed bid projects claimed that fixed bid projects are the reality of the software industry and these projects can greatly benefit from the agile practices. They also agreed that personally fixed bid projects are not the best way to budget a project.

While the rest claimed that fixed bid projects are essentially against the spirit of agile development. According to Kent Beck, there are 4 variables on any project:
1. Scope
2. Time
3. Cost
4. Quality

Agile practitioners believe that change is inevitable. The requirements change, people change, market change, business rules change. Hence we cannot fix all these variables upfront. We would really like to keep the scope under our control even if the other three variables are fixed.

But fixed bid projects by definition expect all the four variables to be fixed. They expect that with good planning and estimation, one can fixed the scope, time and cost. Usually there is no option on quality. Hence everything is fixed upfront.

Some panelist felt that in spirit agile and fixed bid projects are oxymoron. But the apposing panelist felt that since we can use some of the practices and benefit from it, fixed bid agile projects are possible.

The point is just because you are using some of the practices, does it make you agile? One can apply some of the agile practices on a waterfall project; does that mean agile waterfall projects are possible? Nothing in waterfall stops you from pair programming, collective code ownership, refactoring, TDD, etc.

Next time you think of fixed bid agile projects, ask yourself whom are you fooling?

Recently I presented on acceptance testing at the Bangalore Agile User Group meeting.

We started the discussion with an interesting question,
What is different about user acceptance tests in Agile compared to what we do in the traditional model?

Though the concept of user acceptance tests has been around for ages, it somehow does not seem to be really helping. According to me, the main difference is in the approach. In traditional models, user acceptance test is done at the end of the release and it‘s always too late in process to get any constructive feedback.

Since the practice of acceptance tests seems to be really important, in Agile, we try doing it all the time. There are two major differences as I see:
1. We drive our development thru the acceptance tests. It is Test Driven Design [TDD] at a much higher level.
2. There is a heavy focus on automation. Automated acceptance tests really help to express the acceptance criteria in an unambiguous fashion. Frameworks like FIT/Fitnesse really make it possible for the development team and the customers to have concrete discussions using fit tests/documents as examples.

In India, a lot of companies follow the Validation – Verification model or the V model of software development. It is a waterfall model of software development. It starts with requirements analysis and ends with User Acceptance tests. It‘s usually at least 4 to 6 months before we start user acceptance testing. Since we have such a long feedback cycle, I have heard and experienced horror stories with this model.

Some obvious loop holes with this model as far as acceptance testing is concerned are:

• Keeping in sync with requirements specs
• Lack of feedback on the direction of dev with reference to user requirements
• Lack of automation: acceptance tests are not automated and hence very inefficient.
• Lack of testable design
• UAT is deferred till very late in dev cycle
• Productivity hit due to rework since feedback is delayed
• Lack of measure of progress
• Preparing test data model is not a part of the process. There is no clear test data separation.
• User involvement is late or never
• Lot of non-requested or non required features are delivered.
• There is no easy starting point for a new team member on a project. One has to go thru hundreds of requirement and design documents before understanding what the objective of the project is. But they still do not have some examples or code entry points.

Clearly, the agile way of doing acceptance testing can resolve these issues.

1. Keeping in sync with requirements: If we drive the development using acceptance tests, it is very difficult to go out of sync.

2. Lack of feedback: Again, if we are driving development using acceptance tests and running them as a part of every build, you can get instantaneous feedback.

3. Lack of automation: Using frameworks like Fit/Fitnesse helps you automate the acceptance tests with very little effort.

4. UAT is deferred till very late in dev cycle: By driving development thru acceptance tests you doing continuous acceptance testing.

5. Productivity hit due to rework since feedback is delayed: A lot of rework is saved since we get constant feedback from the acceptance tests. Problems detected earlier on are much cheaper to fix compared to them detected later on.

6. Lack of measure of progress: How do you measure progress? According to me there is no real measure of progress other than tested, working features delivered to the customer. But automated acceptance tests do give some useful information on progress. In a given sprint or iteration, if the team plans to develop n number of stories and each story has one or more acceptance test. Then, at any give point one could run the suite of acceptance tests and see how many pass. This can give quite an accurate measure of progress.

7. Preparing test data model is not a part of the process. There is no clear test data separation: Fit/Fitnesse does quite a good job at separating test data from test code. Fit tests can clearly express the intent and abstract the plumbing details from the test data and the intent.

8. User involvement is late or never: Acceptance tests using Fit/Fitnesse is a contract for further discussion with the customer. They provide a good set of examples which the development team can use to discuss domain specific details with the customer.

9. There is no easy starting point for a new team member on a project: Most of the projects I have been on, it‘s very difficult to find a good starting point. First couple of weeks you spend reading a bunch of documents, which are mostly outdated. I would always want to jump into code and see what is happening. Acceptance tests give me that entry point which can define the functionality with concrete examples.

10. Non requested or non required features: Acceptance tests help the team to focus on the tasks/features on hand and only deliver things that are prioritized by the customer.

11. Lack of testable design: Driving development thru acceptance tests lets one develop a design which lends itself naturally to better testability. I have also seen nicely decoupled design with this approach.

I‘m glad I did not have to try too hard to convince the folks about this.

Since majority of the folks were new to acceptance testing, I did a small intro talk and then demonstrated acceptance testing on a sample application that I‘m developing for Agile India website.

Agile India 2006 was the 2nd annual conference on Agile in Bangalore. This year‘s conference had a wider representation of topics. We discussed challenges in implementing XP, we discussed a case study with Scrum, we had a presentation on Agile MDD and an intro talk on DSDM. Apart from these, we also had a wide range of interesting technical topics and some workshops.

Craig Larman gave a great keynote talk discussing some of the myths with traditional software development. On the second day we had a keynote from Steve Messenger on DSDM.

For a detailed schedule please refer to the ASCI website – http://agileindia.org/agileindia2006/program.htm

In one year, it looks like we have come a long way in terms of the topics and the participation from companies. We had participants registered from 73 different companies and 6 Universities from all over India. We also had speakers from 13 different companies.

We had a focused group of organizers [members of ASCI] who spent their after work hours to put this conference together. Hats off to all the organizers for the great work. It‘s incredible to watch things fall into place.

We were targeting around 250 delegates for this conference, but unfortunately only 150 showed up. As a part of the delegate kit we gave the following,
1. Conference T-Shirts,
2. Agile Project Management book by Jim Highsmith or Agile Software Development book by Alistair Cockburn.
3. DVD containing open source projects recommended by ThoughtWorks.

The two days of the conference went on pretty smoothly without any major issues. We concluded the conference with a retrospective which involved all the organizers, volunteers and delegates. The retrospective gave the delegates a hands on session on retrospectives, a platform for everyone to express their thoughts and a great feedback mechanism for the organizers.

Jim Shore was kind enough to work with the group to reschedule the meeting while he is in the area.

Date & Time: May 15th, 2006 from 6:30 PM onwards

Topic: “The Tortoise and the Hare.“

Location:
Fidelity National Information Services (FIS)
2 West Liberty Boulevard
Malvern, PA

Agenda:
1. 6:30 – 7:00 : Greet, intro, and announcements, etc.
2. 7:00 – 8:30 : James Shore http://jamesshore.com will be conducting an activity called “The Tortoise and the Hare.” It is about using tools to speed up agile planning.
3. 8:30 onwards – Discussion

So, if you want to attend, please RSVP via email to nocksj@sourcextreme.com ASAP. Please include your name and organization.