Separate Developers for Functional Testing?
I’ve been working with a Product company for 15 months now. About a year and half ago, this company had decided to hire a separate QA team for doing functional testing of their products. The original idea was to use an army of manual testers. On discussion with me it became clear that this army of manual tests would soon become the bottleneck in their process. Manual testing is not scalable nor sustainable for a large number of products. So they agreed to embrace automation.
Since automated functional testing involves writing code/scripts to test code, it was realized that they need developers to do this job (unless they found a specialist automation tester). Hence they agreed that developers will be responsible for building functionality and also writing automated functional tests. Either a developer does both (development and functional testing) for the same task or the developer finds another developer on the team to write functional tests for this task while s/he writes functional tests for someone else’s task. The developers were responsible for writing unit tests anyway, now they were also responsible for writing automated functional tests.
Unfortunately a year later we don’t see all teams writing functional tests. Some teams have really embraced this and they are doing great, while others are even struggling to keep up with unit testing. We don’t know the exact reasons. Here are some potential issues:
- Some developers complain that they don’t have time to write tests
- While others complain that there is a lack of good tools to do good functional testing.
- Some developers don’t see value in writing functional tests (some developers even look down upon functional tests)
- Some developers are not sold on the concept that its their responsibility to write tests. (Its the job of a QA)
- Knowledge: Developers might have not been exposed to the concept of Functional Tests
So some members of this organization have come up with a proposal to hire 1 or 2 developers per team. The primary responsibility of these developers would be to write automated functional tests. These developers need to be passionate about testing. The required training would be given to those folks to make sure they understand the functionality & technology of the products being building (which they will be testing) and the tools required to do their job. These developers would be part of the team and will report into the tech-lead.
While I’m very skeptical to this idea, I’m willing to give it a try.
Points driving this proposal:
- In the last couple of months, the organization has interviewed few good candidates who don’t qualify the company’s recruitment bar for developers from certain point of view, like technology breath. But these are sharp individuals, eager to learn new stuff and they really want to join the company. By creating this role the company will be able to hire them.
- Since functional testing requires a different thought process, having someone else focus on it can be more effective. (Personally I don’t buy this)
- There is a bandwidth issue with the developers, adding functional testers can ease some of the bandwidth issue. (Based on my experience even after adding functional tests, I think it would still take the same amount of elapsed time to build and test a feature)
- Since someone is now responsible for tests, there is a higher chance that it would be taken care of (tests would be created for new functionality and bugs)
- Having another person test code from a functional stand point might help cover things that a developer might have missed out (another pair of eyes thinking from a different perspective)
Points to seriously consider
- Trying to parallelize development and functional testing to save time is not really effective. Because without thinking about acceptance criteria (tests scenarios) developers can’t build a feature. This means that both the developer and the functional tester have to sit together to come up with scenarios and then do development. Most of the times we find new scenarios during development. In a lot of cases one cannot build functional tests until some portions of the feature is developed.
- Separating concern by letting the developer focus on coding and the tester focus on functional testing could encourage an attitude that testing (read it as Quality) is not my responsibility, let someone else take care of it. Which means the chances of building untestable code is higher. (Its not my problem)
- For someone to be effective at functional testing, that person needs to have, if not better, at least as good an understanding as the developers about functionality and technology. In a lot of cases developers who built the feature with all that knowledge find it difficult to test something. Not sure how effective someone else would be.
- Who will be responsible for coming up with the test plans and scenarios? Are these functional testers really in a position to come up with effective tests? Or will it drain too much time from the tech lead? I hope the goal is not to generate a lot of tests. We want few well crafted tests that give us the confidence we need. Its about quality of tests and not their quantity.
- While developing a feature, as a developer, we do lots of testing of the feature ourselves. Now if someone else is building the automated tests, then the developer will have to do their testing manually. This would be a huge waste of time for the developer. If we are already writing automated tests than, how much value will that newly trained person add? Also later on, when a bug is found, we cannot resort to manual testing there, so developers need to automate these tests anyways.
- At the face of it, it might look like functional testing is a trivial thing, any t-d-h can build it. But the reality is, they can get very complex. Imagine managing integration points, doing the necessary setup and teardown in the test, making sure tests are independent, isolated and can run in parallel. Its not as simple as it might look.
- Some folks might think of functional testing as record-and-play back testing. If you have actually used this approach to testing, you will realize how flawed this technique is. Take a simple scenario in webmail: deleting an email. I can go select a message and click on the delete button and make sure the mail disappears from my inbox and shows up in trash. Now try running this test again. For this test to work, you need a mail with the same id in your inbox. Things can get even more complicated if you have AJAX and stuff like that.
- In my experience, maintaining test cases is a huge effort. It needs really good development skill. Not sure if newly hired developers will have the necessary skill sets
- Once a set of features are built and developed, we might have to make a change, which might break a lot of tests. If a developer has to fix all the tests, s/he might be very careful about this change. May be find a way not be break those tests. But I’ve seen that when it’s not a developer’s responsibility to fix/maintain those tests, people tend to be less careful about breaking stuff. Coz its someone else’s job to maintain it. Its not my problem.
- Frameworks and tools for testing: We don’t have stable and usable frameworks and tools out there for testing. In my experience, in many cases we have to hack existing frameworks/tools or build something from scratch. Those frameworks and tools need to evolve over a period of time. Not just that, but even our tests needs to evolve. Not sure if functional testers will be able to do that.
- When will the functional testers create tests for features that get completed at the end of the iteration? Are we going to have a code freeze couple of days in advance so that testers have time to create functional tests? What if they find bugs during the last couple of days?
- In some cases I’ve seen that testers change (fix) the test, when its actually a problem with the code. Coz functional testers are not confident that they know enough, it is more convenient to change the test than to cause a build failure.
- Like it or not, when someone proves that your code has bugs, person’s ego is hurt. This leads to some tension building up between the 2 people. Developers tend to get defensive. There is lot of back and forth going on between the testers and developers saying “we cannot reproduce the bug” and so forth. Are we ready to handle this? Will this not take more time away?
One needs to pay serious attention to these points before they go down the route of having separate developers for testing. Having a separate QA team is out of question.
