Recently I presented on acceptance testing at the Bangalore Agile User Group meeting.
We started the discussion with an interesting question,
What is different about user acceptance tests in Agile compared to what we do in the traditional model?
Though the concept of user acceptance tests has been around for ages, it somehow does not seem to be really helping. According to me, the main difference is in the approach. In traditional models, user acceptance test is done at the end of the release and it‘s always too late in process to get any constructive feedback.
Since the practice of acceptance tests seems to be really important, in Agile, we try doing it all the time. There are two major differences as I see:
1. We drive our development thru the acceptance tests. It is Test Driven Design [TDD] at a much higher level.
2. There is a heavy focus on automation. Automated acceptance tests really help to express the acceptance criteria in an unambiguous fashion. Frameworks like FIT/Fitnesse really make it possible for the development team and the customers to have concrete discussions using fit tests/documents as examples.
In India, a lot of companies follow the Validation – Verification model or the V model of software development. It is a waterfall model of software development. It starts with requirements analysis and ends with User Acceptance tests. It‘s usually at least 4 to 6 months before we start user acceptance testing. Since we have such a long feedback cycle, I have heard and experienced horror stories with this model.
Some obvious loop holes with this model as far as acceptance testing is concerned are:
- Keeping in sync with requirements specs
- Lack of feedback on the direction of dev with reference to user requirements
- Lack of automation: acceptance tests are not automated and hence very inefficient.
- Lack of testable design
- UAT is deferred till very late in dev cycle
- Productivity hit due to rework since feedback is delayed
- Lack of measure of progress
- Preparing test data model is not a part of the process. There is no clear test data separation.
- User involvement is late or never
- Lot of non-requested or non required features are delivered.
- There is no easy starting point for a new team member on a project. One has to go thru hundreds of requirement and design documents before understanding what the objective of the project is. But they still do not have some examples or code entry points.
Clearly, the agile way of doing acceptance testing can resolve these issues.
1. Keeping in sync with requirements: If we drive the development using acceptance tests, it is very difficult to go out of sync.
2. Lack of feedback: Again, if we are driving development using acceptance tests and running them as a part of every build, you can get instantaneous feedback.
3. Lack of automation: Using frameworks like Fit/Fitnesse helps you automate the acceptance tests with very little effort.
4. UAT is deferred till very late in dev cycle: By driving development thru acceptance tests you doing continuous acceptance testing.
5. Productivity hit due to rework since feedback is delayed: A lot of rework is saved since we get constant feedback from the acceptance tests. Problems detected earlier on are much cheaper to fix compared to them detected later on.
6. Lack of measure of progress: How do you measure progress? According to me there is no real measure of progress other than tested, working features delivered to the customer. But automated acceptance tests do give some useful information on progress. In a given sprint or iteration, if the team plans to develop n number of stories and each story has one or more acceptance test. Then, at any give point one could run the suite of acceptance tests and see how many pass. This can give quite an accurate measure of progress.
7. Preparing test data model is not a part of the process. There is no clear test data separation: Fit/Fitnesse does quite a good job at separating test data from test code. Fit tests can clearly express the intent and abstract the plumbing details from the test data and the intent.
8. User involvement is late or never: Acceptance tests using Fit/Fitnesse is a contract for further discussion with the customer. They provide a good set of examples which the development team can use to discuss domain specific details with the customer.
9. There is no easy starting point for a new team member on a project: Most of the projects I have been on, it‘s very difficult to find a good starting point. First couple of weeks you spend reading a bunch of documents, which are mostly outdated. I would always want to jump into code and see what is happening. Acceptance tests give me that entry point which can define the functionality with concrete examples.
10. Non requested or non required features: Acceptance tests help the team to focus on the tasks/features on hand and only deliver things that are prioritized by the customer.
11. Lack of testable design: Driving development thru acceptance tests lets one develop a design which lends itself naturally to better testability. I have also seen nicely decoupled design with this approach.
I‘m glad I did not have to try too hard to convince the folks about this.
Since majority of the folks were new to acceptance testing, I did a small intro talk and then demonstrated acceptance testing on a sample application that I‘m developing for Agile India website.