Agile FAQs
  About   Slides   Home  

 
Managed Chaos
Naresh Jain's Random Thoughts on Software Development and Adventure Sports
     
`
 
RSS Feed
Recent Thoughts
Tags
Recent Comments

Separate Developers for Functional Testing?

I’ve been working with a Product company for 15 months now. About a year and half ago, this company had decided to hire a separate QA team for doing functional testing of their products. The original idea was to use an army of manual testers. On discussion with me it became clear that this army of manual tests would soon become the bottleneck in their process. Manual testing is not scalable nor sustainable for a large number of products. So they agreed to embrace automation.

Since automated functional testing involves writing code/scripts to test code, it was realized that they need developers to do this job (unless they found a specialist automation tester). Hence they agreed that developers will be responsible for building functionality and also writing automated functional tests. Either a developer does both (development and functional testing) for the same task or the developer finds another developer on the team to write functional tests for this task while s/he writes functional tests for someone else’s task. The developers were responsible for writing unit tests anyway, now they were also responsible for writing automated functional tests.

Unfortunately a year later we don’t see all teams writing functional tests. Some teams have really embraced this and they are doing great, while others are even struggling to keep up with unit testing. We don’t know the exact reasons. Here are some potential issues:

  • Some developers complain that they don’t have time to write tests
  • While others complain that there is a lack of good tools to do good functional testing.
  • Some developers don’t see value in writing functional tests (some developers even look down upon functional tests)
  • Some developers are not sold on the concept that its their responsibility to write tests. (Its the job of a QA)
  • Knowledge: Developers might have not been exposed to the concept of Functional Tests

So some members of this organization have come up with a proposal to hire 1 or 2 developers per team. The primary responsibility of these developers would be to write automated functional tests. These developers need to be passionate about testing. The required training would be given to those folks to make sure they understand the functionality & technology of the products being building (which they will be testing) and the tools required to do their job. These developers would be part of the team and will report into the tech-lead.

While I’m very skeptical to this idea, I’m willing to give it a try.

Points driving this proposal:

  • In the last couple of months, the organization has interviewed few good candidates who don’t qualify the company’s recruitment bar for developers from certain point of view, like technology breath. But these are sharp individuals, eager to learn new stuff and they really want to join the company. By creating this role the company will be able to hire them.
  • Since functional testing requires a different thought process, having someone else focus on it can be more effective. (Personally I don’t buy this)
  • There is a bandwidth issue with the developers, adding functional testers can ease some of the bandwidth issue. (Based on my experience even after adding functional tests, I think it would still take the same amount of elapsed time to build and test a feature)
  • Since someone is now responsible for tests, there is a higher chance that it would be taken care of (tests would be created for new functionality and bugs)
  • Having another person test code from a functional stand point might help cover things that a developer might have missed out (another pair of eyes thinking from a different perspective)

Points to seriously consider

  • Trying to parallelize development and functional testing to save time is not really effective. Because without thinking about acceptance criteria (tests scenarios) developers can’t build a feature. This means that both the developer and the functional tester have to sit together to come up with scenarios and then do development. Most of the times we find new scenarios during development. In a lot of cases one cannot build functional tests until some portions of the feature is developed.
  • Separating concern by letting the developer focus on coding and the tester focus on functional testing could encourage an attitude that testing (read it as Quality) is not my responsibility, let someone else take care of it. Which means the chances of building untestable code is higher. (Its not my problem)
  • For someone to be effective at functional testing, that person needs to have, if not better, at least as good an understanding as the developers about functionality and technology. In a lot of cases developers who built the feature with all that knowledge find it difficult to test something. Not sure how effective someone else would be.
  • Who will be responsible for coming up with the test plans and scenarios? Are these functional testers really in a position to come up with effective tests? Or will it drain too much time from the tech lead? I hope the goal is not to generate a lot of tests. We want few well crafted tests that give us the confidence we need. Its about quality of tests and not their quantity.
  • While developing a feature, as a developer, we do lots of testing of the feature ourselves. Now if someone else is building the automated tests, then the developer will have to do their testing manually. This would be a huge waste of time for the developer. If we are already writing automated tests than, how much value will that newly trained person add? Also later on, when a bug is found, we cannot resort to manual testing there, so developers need to automate these tests anyways.
  • At the face of it, it might look like functional testing is a trivial thing, any t-d-h can build it. But the reality is, they can get very complex. Imagine managing integration points, doing the necessary setup and teardown in the test, making sure tests are independent, isolated and can run in parallel. Its not as simple as it might look.
    • Some folks might think of functional testing as record-and-play back testing. If you have actually used this approach to testing, you will realize how flawed this technique is. Take a simple scenario in webmail: deleting an email. I can go select a message and click on the delete button and make sure the mail disappears from my inbox and shows up in trash. Now try running this test again. For this test to work, you need a mail with the same id in your inbox. Things can get even more complicated if you have AJAX and stuff like that.
  • In my experience, maintaining test cases is a huge effort. It needs really good development skill. Not sure if newly hired developers will have the necessary skill sets
  • Once a set of features are built and developed, we might have to make a change, which might break a lot of tests. If a developer has to fix all the tests, s/he might be very careful about this change. May be find a way not be break those tests. But I’ve seen that when it’s not a developer’s responsibility to fix/maintain those tests, people tend to be less careful about breaking stuff. Coz its someone else’s job to maintain it. Its not my problem.
  • Frameworks and tools for testing: We don’t have stable and usable frameworks and tools out there for testing. In my experience, in many cases we have to hack existing frameworks/tools or build something from scratch. Those frameworks and tools need to evolve over a period of time. Not just that, but even our tests needs to evolve. Not sure if functional testers will be able to do that.
  • When will the functional testers create tests for features that get completed at the end of the iteration? Are we going to have a code freeze couple of days in advance so that testers have time to create functional tests? What if they find bugs during the last couple of days?
  • In some cases I’ve seen that testers change (fix) the test, when its actually a problem with the code. Coz functional testers are not confident that they know enough, it is more convenient to change the test than to cause a build failure.
  • Like it or not, when someone proves that your code has bugs, person’s ego is hurt. This leads to some tension building up between the 2 people. Developers tend to get defensive. There is lot of back and forth going on between the testers and developers saying “we cannot reproduce the bug” and so forth. Are we ready to handle this? Will this not take more time away?

One needs to pay serious attention to these points before they go down the route of having separate developers for testing. Having a separate QA team is out of question.

  • It’s funny because I just wrote a blog post about a team that sounds like the ones who aren’t writing functional tests. The problem is that the whole “don’t have enough time” excuse is indicative that someone in the process is not sold on the value of the functional tests.

    It’s not clear whether that’s the PM pushing to keep getting more and more features in, or whether that’s the developers, who take it on themselves to say there is no time for tests.

    Of course, there IS time to do the tests, it’s right. There always is. That time is right AFTER you’ve completed the feature and right BEFORE you start a new feature. The problem comes in when developers don’t do this and try to push tests in right before a release or something.

    Unfortunately,I think this new setup will struggle just like the current situation. New hires will not have adequate understanding to write good functional tests. In my experience, entry level tests are quite brittle and perfunctory. 
    Ultimately, developers must take full responsibility for automated unit AND functional testing, otherwise quality leaks will continue to happen.  Personally I have appreciated ADDITIONAL manual testing as a just-in-case measure, but that in no way alleviates the developer’s responsibility to confirm the correctness of his own code.

  • It’s funny because I just wrote a blog post about a team that sounds like the ones who aren’t writing functional tests. The problem is that the whole “don’t have enough time” excuse is indicative that someone in the process is not sold on the value of the functional tests.

    It’s not clear whether that’s the PM pushing to keep getting more and more features in, or whether that’s the developers, who take it on themselves to say there is no time for tests.

    Of course, there IS time to do the tests, it’s right. There always is. That time is right AFTER you’ve completed the feature and right BEFORE you start a new feature. The problem comes in when developers don’t do this and try to push tests in right before a release or something.

    Unfortunately,I think this new setup will struggle just like the current situation. New hires will not have adequate understanding to write good functional tests. In my experience, entry level tests are quite brittle and perfunctory. 
    Ultimately, developers must take full responsibility for automated unit AND functional testing, otherwise quality leaks will continue to happen.  Personally I have appreciated ADDITIONAL manual testing as a just-in-case measure, but that in no way alleviates the developer’s responsibility to confirm the correctness of his own code.

  • D. André Dhondt

    In my opinion, 2 new hires will never be able to keep up with the functional testing needs of even a small group of developers.  If code isn’t written and designed from the start to be testable, it is MUCH more expensive to test, the tests are much less reliable, and you end up with a vicious cycle (these test’s aren’t reliable, so we have to test manually, they’re not reliable, so we shouldn’t add more tests, etc.)  I think this plan is a half-hearted way to say that testing isn’t very important.  Get the priorities straight or you can’t win.

  • D. André Dhondt

    In my opinion, 2 new hires will never be able to keep up with the functional testing needs of even a small group of developers.  If code isn’t written and designed from the start to be testable, it is MUCH more expensive to test, the tests are much less reliable, and you end up with a vicious cycle (these test’s aren’t reliable, so we have to test manually, they’re not reliable, so we shouldn’t add more tests, etc.)  I think this plan is a half-hearted way to say that testing isn’t very important.  Get the priorities straight or you can’t win.

  • I don’t think, a parallel QA team is a good solution to write functional tests.

    Functional testing can not be looked in separation as Testing code needs the same love and care as developement code.

    My suggestion

    – Have a team of QA Functional testers, who would work in parallel with devs team to make sure Functional testing is done in a a right way, Dev bandwidth is consumed in the right way to help them writing the tests and these functional testers get the right technical help from the devs if needed.

    – Automation testing can not be succeded without the collabration with developers. As they are the people who have the first control over the quality of the code.

  • I don’t think, a parallel QA team is a good solution to write functional tests.

    Functional testing can not be looked in separation as Testing code needs the same love and care as developement code.

    My suggestion

    – Have a team of QA Functional testers, who would work in parallel with devs team to make sure Functional testing is done in a a right way, Dev bandwidth is consumed in the right way to help them writing the tests and these functional testers get the right technical help from the devs if needed.

    – Automation testing can not be succeded without the collabration with developers. As they are the people who have the first control over the quality of the code.

  • I’ve seen more value in a developer and a tester pairing to write automated functional tests. This provides both the automation expertise and the testing viewpoint.

    Some testing tools, such as fitnesse and cucumber, are easy for a tester to add additional scenarios once the lower level fixture has been programmed so that it drives the application.

    Side note: Amber, you say, “time [to write tests] is right AFTER you’ve completed the feature and right BEFORE you start a new feature.” I disagree. I find a better time is at the SAME time as you start the feature. Then the tests can be used as a goal for the developer, avoiding the tension of “your code has bugs.” This puts the developers and the testers on the same side, working to provide a correct application, rather than in opposition.

  • I’ve seen more value in a developer and a tester pairing to write automated functional tests. This provides both the automation expertise and the testing viewpoint.

    Some testing tools, such as fitnesse and cucumber, are easy for a tester to add additional scenarios once the lower level fixture has been programmed so that it drives the application.

    Side note: Amber, you say, “time [to write tests] is right AFTER you’ve completed the feature and right BEFORE you start a new feature.” I disagree. I find a better time is at the SAME time as you start the feature. Then the tests can be used as a goal for the developer, avoiding the tension of “your code has bugs.” This puts the developers and the testers on the same side, working to provide a correct application, rather than in opposition.

  • Sounds like a complex problem. My first reaction is, why aren’t the teams using the tests to drive development? Testing and coding ought to be one process, instead of thinking of testing as some phase that comes later.

    You still will need some post-development exploratory testing, but if the focus is on quality and driving with tests, it should change the developers’ mindset.

  • Sounds like a complex problem. My first reaction is, why aren’t the teams using the tests to drive development? Testing and coding ought to be one process, instead of thinking of testing as some phase that comes later.

    You still will need some post-development exploratory testing, but if the focus is on quality and driving with tests, it should change the developers’ mindset.

  • Tungano

    Typically a profession doesn’t have ‘time’, if they do they are a resource wasted. Not having time is really a way of saying “I find other things more important” or “My priorities lay elsewhere”.

    I think your post has plenty of solid arguments to let developers write tests themselves. One could argue letting them feel the ‘pain’ of manual testing would help encourage them. Perhaps seeing testing as a developer discipline and not as a separate job function is the way to go…

  • Tungano

    Typically a profession doesn’t have ‘time’, if they do they are a resource wasted. Not having time is really a way of saying “I find other things more important” or “My priorities lay elsewhere”.

    I think your post has plenty of solid arguments to let developers write tests themselves. One could argue letting them feel the ‘pain’ of manual testing would help encourage them. Perhaps seeing testing as a developer discipline and not as a separate job function is the way to go…


    Licensed under
Creative Commons License