» Test Automation Dilemma » Managed Chaos by Naresh Jain

Test Automation Dilemma

I regularly practice Test Driven Development. I truly believe in its benefits and also have been able to influence a huge number of develops to practice it.

However, there are situations in which I ask myself questions like:

Is it important to automate this particular check (test) ?
Will it be worth the effort/investment or is it YAGNI?
What is the possibility of this particular scenario breaking (risk)?
And so on…

Yesterday I was faced with a similar situation where I decided to skip the automated test.

Context: I was working on a website where users can use their credit card while shopping online. Basically the website had a simple text input box to accept the credit card number. It turns out that most browsers (except Safari) caches the input data and stores it in plain text somewhere on the computer.

So we wanted to turn off the auto-complete and caching feature on this specific input field. There is a very easy way to do this. Just need to set autocomplete=”off” in the input tag. For example:

Its an easy fix and quite widely used. So that’s great. But how do I write an automated test for this?

If we think a little hard, there are ways to write automated test. But then you ask yourself is it worth it?

This site had lived without this feature for so long, so I did not think it was that crucial to its users. Even if this feature stops works, it won’t bring the site down. (They certainly had a good battery of automated tests which tests the core functionality if the product.) So I choose to skip the automated test. I manually tested it with different browsers made sure it was working as expected.

If this comes back and bites me, I’ll certainly invest in some form of safety net, but for now, time to hack some more code.

What would you choose to do?

This entry was posted on Thursday, June 24th, 2010 at 6:15 PM and is filed under Agile, Testing. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

http://agileshrugged.com Nayan Hajratwala

Since you know that autocomplete=”off” does the trick, how about a simple unit test that takes a look at all your html files/templates and ensures that any field with a name of “creditcard” has that attribute set appropriately?
Chris Dams

Well, outputting HTML is essentially the user interface and that is not so convenient (or usual) to do TDD-tests on that. You could do what Nayan said, though and it makes sense to do that. Hopefully, “html files/templates” is not applicable, though…. The refactoring part of TDD should do away with those. Html files/templates are nasty nests of code duplication. I believe all HTML should be generated by code and not be in files and/or templates. Having it in templates suffers at that from the disease of having to invent your own programming language.
Graham

I have to confess that if I already had an acceptance test checking key attributes of elements on that page I would have added the test. Although it does not bring down the site not caching the value talks to the information security of the site. People who have used it before might be alarmed if the attribute changes and their data is stored locally.
Laurent Bossavit

On top of a “safety net” against regression, a test buys you some self-documenting properties for your code: it records the reasons for which a credit card input should not auto-complete, and states what the correct behaviour is in such cases.

My focus is generally very much on the four Rules of Simplicity: 1) all the tests pass, 2) every distinct idea has a name, 3) every idea is expressed only once, 4) the number of entities is minimal.

You've ended up with a small gap in the second rule. The idea that “input fields for sensitive data should not autocomplete because that entails retention of the data in browser caches” exists only in the programmer's mind, and the code manifests this idea implicitly, not explicitly.

It's not too important what kind of test records this concern – Nayan has a point, a unit test would be enough – what is important is that the concern should be recorded.
http://blogs.agilefaqs.com Naresh Jain

Thanks Nayan, Chris, Graham and Laurent. Those are some great suggestions.
jdtangney

I remember discussing this with you at time. I felt great impatience. I wanted to just say “Forget it. Life's too short.” I was struggling with how to emulate the auto-fill in a test environment. I was making it too complicated!

I had completely forgotten about another scenario that easily reproduces the problem, and makes an automated test almost trivial.

So trivial, in fact, that it was there already! The scenario in question was as follows:
1) Fill in the form
2) Submit
3) Go back to the form (browser's back button)
The fields should all have their previous values – except for the cc field which should now be empty. All we had to do was modify the existing test to verify that the cc field is indeed empty, thereby guarding against regression and (more importantly in this case) documenting the expected behavior.

But the larger point – how much is “enough” automated testing – is an issue that I have tended to face using instinct alone while weighing RoI. I like Laurent's 4 Rules of Simplicity. But I think there are some deeper and more specific heuristics we can codify.

In fact, just today I was looking at a massive combinatorial explosion of environments in which some code needs to run. Automation! Of course! Why would anyone NOT want to automate it?

Here's why not: It will take days to set it up, will require hardware resources we don't have, and will distract us from more urgent, revenue-producing efforts.

So what's the alternative? Test it by hand, but for goodness sake, test it!
http://jdtangney.com jdtangney

I remember discussing this with you at time. I felt great impatience. I wanted to just say “Forget it. Life's too short.” I was struggling with how to emulate the auto-fill in a test environment. I was making it too complicated!

I had completely forgotten about another scenario that easily reproduces the problem, and makes an automated test almost trivial.

So trivial, in fact, that it was there already! The scenario in question was as follows:
1) Fill in the form
2) Submit
3) Go back to the form (browser's back button)
The fields should all have their previous values – except for the cc field which should now be empty. All we had to do was modify the existing test to verify that the cc field is indeed empty, thereby guarding against regression and (more importantly in this case) documenting the expected behavior.

But the larger point – how much is “enough” automated testing – is an issue that I have tended to face using instinct alone while weighing RoI. I like Laurent's 4 Rules of Simplicity. But I think there are some deeper and more specific heuristics we can codify.

In fact, just today I was looking at a massive combinatorial explosion of environments in which some code needs to run. Automation! Of course! Why would anyone NOT want to automate it?

Here's why not: It will take days to set it up, will require hardware resources we don't have, and will distract us from more urgent, revenue-producing efforts.

So what's the alternative? Test it by hand, but for goodness sake, test it!
http://www.air-jordan-20.com air jordan 20

“Here air jordan 21 products xx, has fashion model, superior quality and service, cheap price and updates quickly.I support strongly always! I want to buy XX, I hesitate to select which style more better.Hope your unique recommends.
”