Most software organizations today suffer from what I called the “Inverted Testing Pyramid”. They spend maximum time and effort building end-to-end GUI test. Very little effort is spent on building unit/micro tests. Hence they end up with majority (80-90%) of their tests being end-to-end GUI tests. Some effort is spent on writing so-called “Integration test” (typically 5-15%.) Resulting in a shocking 1-5% of their tests being unit/micro tests.

Why is this a problem?

• The base of the pyramid is constructed from end-to-end GUI test, which are famous for their fragility and complexity. A small pixel change in the location of a UI component can result in test failure. GUI tests are also very time-sensitive, sometimes resulting in random failure (false-negative.)
• To make matters worst, most teams struggle automating their end-to-end tests early on, which results in huge amount of time spent in manual regression testing. Its quite common to find test teams struggling to catch up with development. This lag causes many other hard-development problems.
• Number of end-to-end tests required to get a good coverage is much higher and more complex than the number of unit tests + selected end-to-end tests required. (BEWARE: Don’t be Seduced by Code Coverage Numbers)

• Maintain a large number of end-to-end tests is quite a nightmare for teams. Following are some core issues with end-to-end tests:
  • It requires deep domain knowledge and high technical skills to write quality end-to-end tests.
  • They take a lot of time to execute.
  • They are relatively resource intensive.
  • Testing negative paths in end-to-end tests is very difficult (or impossible) compared to lower level tests.
  • When an end-to-end test fails, we don’t get pin-pointed feedback about what went wrong.
  • They are more tightly coupled with the environment and have external dependencies, hence fragile. Slight changes to the environment can cause the tests to fail. (false-negative.)
  • From a refactoring point of view, they don’t give the same comfort feeling to developers as unit tests can give.

Again don’t get me wrong. I’m not suggesting end-to-end integration tests are a scam. I certainly think they have a place and time.

What I propose and help many organizations achieve is the right balance of end-to-end tests, acceptance tests and unit tests. I call this “Inverting the Testing Pyramid.” [Inspired by Jonathan Wilson's book called Inverting The Pyramid: The History Of Football Tactics].

In a later blog post I can quickly highlight various tactics used to invert the pyramid.

Simple Design and Testing Conference is an all open space conference providing software practitioners a platform to meet face-to-face and discuss/demonstrate simple design & testing principles/approaches.

At this conference you’ll meet real, hands-on practitioners interested in peer-to-peer learning and exploration. We strive hard to avoid fluffy, marketing talks and other non-sense.

• What: Open Space Conference on Simple Design & Testing practices
• Where: Skills Matter eXchange, London, UK
• When: 12th-13th Mar 2011
• Who: Software Practitioners (Developers, Testers, UX Designer…)
• Cost: £50.00, also (Position Paper required!)

SDT Conf 2011 is our 6th annual conference and for the first time in Europe. Check out the past conference SDT Conf 2006, 2007, 2008, 2009 and 2010 details.

Register now…

You are telling me, if I were unit testing a List class and I wrote tests like the following with 2 assert statements in them, then I’m violating xUnit testing “best practices”.

@Test
public void addingElementIncreasesSizeOfTheList() {
	assertEquals(0, list.size()); 
       //or assertListSizeIs(0); - define your custom assert statement
	list.add("Element");
	assertEquals(1, list.size()); // assertListSizeIs(1);
}

If so, how and why?

I understand that if I wrote a test like the following its not very communicative:

@Test
public void removeElementBasedOnItsIndexAndReduceSizeByOne() {
	list.add("Element");
	assertEquals("Element", list.remove(0));
	assertEquals(0, list.size());
}

in this case, it might be better to write 2 test instead:

@Test
public void removeElementBasedOnItsIndex() {
	list.add("Element");
	assertEquals("Element", list.remove(0));
}
 
@Test
public void reducesSizeByOneOnRemovingAnElement() {
	list.add("Element");
	assertEquals(1, list.size());
	list.remove(0);
	assertEquals(0, list.size());
}

Notice our goal was better communication and one of the tests we ended up (incidentally) with had just one assert statement in it (which is better than the previous test which was asserting 2 different things).

Our goal is not one assert statement per test. Our goal is better communication/documentation. Using the one-assert-per-test heuristic helps to achieve better communication. But splitting tests into more tests or deleting an important assert statement from a test just to avoid multiple assert statements in a test is mindless dogma.

Conclusion: One-Assert-Per-Test really means test one aspect/behavior in each test (and communicate that clearly in your test name). It does not literally mean, one test should have only one assert statement in it. Test one aspect/behavior in one test and related aspect/behavior can be tested in another test. Sometimes we need more than one assert statement to assert one aspect/behavior and its helps to add those assert statements in the test for better communication.

So don’t be afraid, go ahead and add that second or third assert statement in your test. Be careful not to fill your test with many asserts, it becomes difficult to understand and debug.

UPDATE:
Folks, I’m sorry for using a naive code example. I agree the example does not do justice to the point I’m trying to make.

@Robert, I certainly like your test. Much better.

Here is another naive example, looking forward to brickbats

Let’s assume I’m building a RomanDigit class and I need to make sure that RomanDigits are map friendly, .i.e. they work correctly when added to maps. Following is a test for the same:

@Test
public void isMapFriendly() throws Exception {
	RomanDigit one = new RomanDigit('I', 1);
	RomanDigit anotherOne = new RomanDigit('I', 1);
	Map romanDigits = new HashMap();
	romanDigits.put(one, "One");
	romanDigits.put(anotherOne, "AnotherOne");
	assertEquals(1, romanDigits.size());
	assertEquals("AnotherOne", romanDigits.get(one));
	assertEquals("AnotherOne", romanDigits.get(anotherOne));
}

Another example: When subtracting RomanDigits, only powers of ten can repeat.

@Test
public void onlyPowerOfTenCanRepeate() throws Exception {
	assertTrue("I is a power of Ten and can repeat", romanDigit('I').canRepeat());
	assertFalse("V is not a power of Ten and should not repeat", romanDigit('V').canRepeat());
}

Is there a better way to write these tests so that I only need one assert statement per test?

I’m proud to announce the 5th Annual SDTConf. This year we plan to hold the conference in Otterbein University Campus Center, OH, USA.

We plan to keep a max cap of 100 participants for this conference.

As you might be aware SDTConf is a free conference and we use the concept of position papers as the price for admission. This helps us ensure the quality of the participants is really high. You can add your position papers for the conference on our wiki. Making the position papers public helps other participants gauge in advance what they can expect from the conference.

Last but not the least, since this is a community run, non-profit event, we really on sponsorship in kind to make this event possible. Here is a list of items that you or your company can sponsor to support this conference.

P.S: Please blog about this conference and/or send an email to your friends and colleagues. Word of mouth is the only way we market this event.

I regularly practice Test Driven Development. I truly believe in its benefits and also have been able to influence a huge number of develops to practice it.

However, there are situations in which I ask myself questions like:

• Is it important to automate this particular check (test) ?
• Will it be worth the effort/investment or is it YAGNI?
• What is the possibility of this particular scenario breaking (risk)?
• And so on…

Yesterday I was faced with a similar situation where I decided to skip the automated test.

Context: I was working on a website where users can use their credit card while shopping online. Basically the website had a simple text input box to accept the credit card number. It turns out that most browsers (except Safari) caches the input data and stores it in plain text somewhere on the computer.

So we wanted to turn off the auto-complete and caching feature on this specific input field. There is a very easy way to do this. Just need to set autocomplete=”off” in the input tag. For example:

<input type=”TEXT” name=”creditcard” autocomplete=”off” />

Its an easy fix and quite widely used. So that’s great. But how do I write an automated test for this?

If we think a little hard, there are ways to write automated test. But then you ask yourself is it worth it?

This site had lived without this feature for so long, so I did not think it was that crucial to its users. Even if this feature stops works, it won’t bring the site down. (They certainly had a good battery of automated tests which tests the core functionality if the product.) So I choose to skip the automated test. I manually tested it with different browsers made sure it was working as expected.

If this comes back and bites me, I’ll certainly invest in some form of safety net, but for now, time to hack some more code.

What would you choose to do?

This post is moved to Industrial Logic’s Blogic.

Mike Bria of InfoQ interviewed me yesterday to find out what is happening with SDTConf. Here is the interview notes on InfoQ. Enjoy!

The First Simple Design and Testing Conference in India is only 2 weeks away. There is lot of excitement, but we’ve not seen many position papers come in, so far. If you plan to attend, please submit your position paper on the conference wiki before 21st June.

I’m planning the Simple Design and Testing Conference (SDTConf) for the first time in India around end of June.

SDTConf is:

• All Practitioner conference (No Jokers giving lectures)
• All Open Space (discussions and lots of hands-on workshops)
• Invitation Only Conference
• I’ve organized this conference in the US for the last 4 years.

For Freeset, I’ve always been in the quest of Simplest Thing that Could Possibly Work. In a previous post, I explained how we’ve embraced an ultra-light process (call it lean, if you like) to build their e-commerce site.

In that post, I’ve talked about our wish to create a Selenium test suite for regression testing. But it never got high enough on our priority list. (esp. coz we mostly have static content served from a CMS as of now).

While that is something I wanted to tackle, last night, when I was moving Industrial Logic and Industrial XP‘s site over to a new server hardware, I wanted some quick way to test if all the pages were correctly displayed after the move. This was important since we switched from Apache to Nginx. Nginx has slightly different way to handle secure pages, etc.

So I asked on Twitter, if anyone knew of a tool that could compare 2 deployments of the same website. Few people responding saying I could use curl/wget with diff recursively. That seemed like the simplest thing that could work for now. So this morning I wrote a script.

rm -Rf * && mkdir live && cd live && wget -rkp -l5 -q -np -nH http://freesetglobal.com && cd .. && mkdir dev && cd dev && wget -rkp -l5 -q -np -nH http://dev.freesetglobal.com && cd .. && for i in `grep -l dev.freesetglobal.com \`find ./dev -name '*'\`` ; do sed -e 's/dev.freesetglobal.com/freesetglobal.com/g' $i > $i.xx && mv $i.xx $i; done && diff -r -y --suppress-common-lines -w -I '^.*' dev live

I’m planning to use this script to do simple regression test of our Freeset site. We have a live and a dev environment. We make changes on dev and frequently sync it up with live. I’m thinking before we sync up, we can check if we’ve made the correct changes to the intended pages. If some other pages show up in this diff that we did not expect, it’s a good way to catch such issue before the sync.

Note: One could also use diff with -q option, if all they are interested to know is which pages changes. Also note that under Mac, the sed command’s -i (inline edit) option is broken. It simply does not work as explained. If you give sed -i -e …., it ends up creating backup files with -e extension. #fail.