Agile FAQs
  About   Slides   Home  

 
Managed Chaos
Naresh Jain's Random Thoughts on Software Development and Adventure Sports
     
`
 
RSS Feed
Recent Thoughts
Tags
Recent Comments

Value of End to End Functional Checks

At the Agile 2009 conference I attended a workshop by Alro Belshee and Jim Shores on “Slow and Brittle: Replacing End to End Tests“. In this workshop they presented a couple of cases where End-to-End checking (testing) became a huge issues as the tests became slow, brittle and expensive. While I agree with them and have seen cases of checks (not just end-to-end checks) becoming a maintenance nightmare. Many times, checks create an illusion of safety, but really don’t add much value.

While some of it can be attributed to the poor scope/focus selection for checking (end-to-end or unit), I also think a lot has to do with poor skills, lack of understanding and rigid/flawed organizational structures.

During the workshop, the group I participated in, came up with following points as reasons why people write end-to-end checks:

  • Document the interaction and intent
  • Being Inclusive: Testers and Programmers can collaborate
  • Scaffolding
    • Good starting point for dealing with legacy code
    • Early win: Easy to show people how to automate checking and get them started
    • Good starting point to learn/explore more about the system
  • Freedom and Confidence to Refactor code
  • Ease: (sometimes they are very easy to create)
  • Highlight Design Problems: If you see duplication in your tests, sometimes it could mean poor design.

I know other teams had several other interesting points. So the question is what other approach can you use to do away with end-to-end checking?

One of my proposals was to look more serious at “Eating your own dog food“. I also very much digg the idea about “Self-learning and Self-testing systems”.

  • http://javadots.blogspot.com/ Itay Maman

    To me, the #1 reason in favor of end-to-end testing is that they provide a very good coverage/effort ratio. In other words: a small amount of end-to-end tests can go a long way in terms of bugs prevention.

  • http://javadots.blogspot.com/ Itay Maman

    To me, the #1 reason in favor of end-to-end testing is that they provide a very good coverage/effort ratio. In other words: a small amount of end-to-end tests can go a long way in terms of bugs prevention.

  • http://www.silverstripesoftware.com/blog/ Siddharta

    Great Post. I think a lot of people jump into automated end to end testing without realising the implications. Things are nice and rosy at the start, but unless you are prepared, it becomes complex later on as you run into severe UI dependencies and slow, brittle tests.

    Thats not to say that these tests are bad. We use them extensively because they do serve a purpose that other tests dont. We can test the app in different browsers, through the UI which is very valuable. But you need to be aware of the limitations and plan for it.

    Problem: Slow – Eventually it is going to be too slow to run the whole suite.
    Solution: Have a way to easily select subparts of the suite to run. Have a way to run tests in parallel.

    Problem: Maintenance – Tests are too dependent on the UI
    Solution: Use good coding practices when writing automated test code. We have a framework which abstracts the test actions to be performed from the UI implementation, so when a UI element changes we change one place and the tests work.

    And whatever happens, never ever use record-replay programs for creating automated tests. You will have unmaintainable scripts that will give you headaches for months to come. We manually code each test.

  • http://www.silverstripesoftware.com/blog/ Siddharta

    Great Post. I think a lot of people jump into automated end to end testing without realising the implications. Things are nice and rosy at the start, but unless you are prepared, it becomes complex later on as you run into severe UI dependencies and slow, brittle tests.

    Thats not to say that these tests are bad. We use them extensively because they do serve a purpose that other tests dont. We can test the app in different browsers, through the UI which is very valuable. But you need to be aware of the limitations and plan for it.

    Problem: Slow – Eventually it is going to be too slow to run the whole suite.
    Solution: Have a way to easily select subparts of the suite to run. Have a way to run tests in parallel.

    Problem: Maintenance – Tests are too dependent on the UI
    Solution: Use good coding practices when writing automated test code. We have a framework which abstracts the test actions to be performed from the UI implementation, so when a UI element changes we change one place and the tests work.

    And whatever happens, never ever use record-replay programs for creating automated tests. You will have unmaintainable scripts that will give you headaches for months to come. We manually code each test.


    Licensed under
Creative Commons License