“Release Early, Release Often” is a proven mantra, but what happens when you push this practice to it’s limits? .i.e. deploying latest code changes to the production servers every time a developer checks-in code?

At Industrial Logic, developers are deploying code dozens of times a day, rapidly responding to their customers and reducing their “code inventory”.

This talk will demonstrate our approach, deployment architecture, tools and culture needed for CD and how at Industrial Logic, we gradually got there.

Process/Mechanics

This will be a 60 mins interactive talk with a demo. Also has a small group activity as an icebreaker.

Key takeaway: When we started about 2 years ago, it felt like it was a huge step to achieve CD. Almost a all or nothing. Over the next 6 months we were able to break down the problem and achieve CD in baby steps. I think that approach we took to CD is a key take away from this session.

Talk Outline

1. Context Setting: Need for Continuous Integration (3 mins)
2. Next steps to CI (2 mins)
3. Intro to Continuous Deployment (5 mins)
4. Demo of CD at Freeset (for Content Delivery on Web) (10 mins) – a quick, live walk thru of how the deployment and servers are set up
5. Benefits of CD (5 mins)
6. Demo of CD for Industrial Logic’s eLearning (15 mins) – a detailed walk thru of our evolution and live demo of the steps that take place during our CD process
7. Zero Downtime deployment (10 mins)
8. CD’s Impact on Team Culture (5 mins)
9. Q&A (5 mins)

Target Audience

• CTO
• Architect
• Tech Lead
• Developers
• Operations

Context

Industrial Logic’s eLearning context? number of changes, developers, customers , etc…?

Industrial Logic’s eLearning has rich multi-media interactive content delivered over the web. Our eLearning modules (called Albums) has pictures & text, videos, quizes, programming exercises (labs) in 5 different programming languages, packing system to validate & produce the labs, plugins for different IDEs on different platforms to record programming sessions, analysis engine to score student’s lab work in different languages, commenting system, reporting system to generate different kind of student reports, etc.

We have 2 kinds of changes, eLearning platform changes (requires updating code or configuration) or content changes (either code or any other multi-media changes.) This is managed by 5 distributed contributors.

On an average we’ve seen about 12 check-ins per day.

Our customers are developers, managers and L&D teams from companies like Google, GE Energy, HP, EMC, Philips, and many other fortune 100 companies. Our customers have very high expectations from our side. We have to demonstrate what we preach.

Learning outcomes

• General Architectural considerations for CD
• Tools and Cultural change required to embrace CD
• How to achieve Zero-downtime deploys (including databases)
• How to slice work (stories) such that something is deployable and usable very early on
• How to build different visibility levels such that new/experimental features are only visible to subset of users
• What Delivery tests do
• You should walk away with some good ideas of how your company can practice CD

Slides from Previous Talks

Recently we realized that our server logs were showing ‘null’ for all HTTP request parameters values.

SEVERE: Attempting to get user with null userName
parameters=[version:null][sessionId:null][userAgent:null][requestedUrl:null][queryString:null]
[action:null][session:null][userIPAddress:null][year:null][path:null][user:null]

On digging around a bit, we found the following buggy code in our custom Map class, which was used to hold the request parameters:

@Override
public String toString() {
    StringBuilder parameters = new StringBuilder();
    for (Map.Entry<String, Object> entry : entrySet())
        parameters.append("[")
                .append(entry.getKey())
                .append(":")
                .append(get(entry.getValue()))
                .append("]");
    return parameters.toString();
}

When we found this, many team members’ reaction was:

If the author had written unit tests, this bug would have been caught immediately.

Others responded saying:

But we usually don’t write tests for toString(), Getters and Setters. We pragmatically choose when to invest in unit tests.

As all of this was taking place, I was wondering, why in the first place, the author even wrote this code? As you can see from the following snippet, Maps already know how to print themselves.

@Test
public void mapKnowsHowToPrintItself() {
    Map hashMap = new HashMap();
    hashMap.put("Key1", "Value1");
    hashMap.put("Key2", "Value2");
    System.out.println(hashMap);
}

Output: {Key2=Value2, Key1=Value1}

Its easy to fall into the trap of first writing useless code and then defending it by writing more useless tests for it.

I’m a lazy developer and I always strive real hard to write as little code as possible. IMHO real power and simplicity comes from less code, not more.

Every single line of code must be unit tested!

This sound advice rather seems quite extreme to me. IMHO a skilled programmer pragmatically decides when to invest in unit testing.

After practicing (automated) unit testing for over a decade, I’m a strong believer and proponent of automated unit testing. My take on why developers should care about Unit Testing and TDD.

However over the years I’ve realized that automated unit tests do have four, very important, costs associated with them:

• Cost of writing the unit tests in the first place
• Cost of running the unit tests regularly to get feedback
• Cost of maintaining and updating the unit tests as and when required
• Cost of understanding other’s unit tests

Why should developers care of automated unit tests?

• Keeps you out of the (time hungry) debugger!
• Reduces bugs in new features and in existing features
• Reduces the cost of change
• Improves design
• Encourages refactoring
• Builds a safety net to defend against other programmers
• Is fun
• Forces you to slow down and think
• Speeds up development by eliminating waste
• Reduces fear

And how TDD takes it to the next level?

• Improves productivity by
• minimizing time spent debugging
• reduces the need for manual (monkey) checking by developers and tester
• helping developers to maintain focus
• reduce wastage: hand overs
• Improves communication
• Creating living, up-to-date specification
• Communicate design decisions
• Learning: listen to your code
• Baby steps: slow down and think
• Improves confidence
• Testable code by design + safety net
• Loosely-coupled design
• Refactoring

Following is a stripped down version of the actual code.

Start with a simple Test:

private static void ContinueMethod(object sender, EventArgs arg)
{
    ((Eventful)sender).ContinueExecutionFor("ContinueMethod");
}
 
[Test]
public void EventHandlerCanRequestFurtherCodeExecution()
{
    var eventful = new Eventful();
    eventful.ContinuationHandler += ContinueMethod;
    eventful.Execute();
    Assert.AreEqual("ContinueMethod wants to execute more logic", eventful.State);
}

and then wrote the production code:

public class Eventful
{
    public string State;
    public event EventHandler<EventArgs> ContinuationHandler;
 
    public void Execute()
    {
        // logic to get ready
 
        var args = new EventArgs();
        //stuff args with some data
        ContinuationHandler(this, args); 
    }
 
    public void ContinueExecutionFor(string delegateName)
    {
        State = delegateName + " wants to execute more logic";
        // some logic
    }
}

The wrote the next Test:

private static void StopMethod(object sender, EventArgs arg)
{
    // no call back
}
 
[Test]
public void EventHandlerCanStopFurtherCodeExecution()
{
    eventful.ContinuationHandler += StopMethod;
    eventful.Execute();
    Assert.IsNull(eventful.State);
}

and this went Green.

Then I wondered what happened if there were no registered event handlers. So I wrote this test.

[Test]
public void NoOneIsInterestedInThisEvent()
{
    eventful.Execute();
    Assert.IsNull(eventful.State);
}

This resulted in:

System.NullReferenceException : Object reference not set to an instance of an object.
at Eventful.Execute() in Eventful.cs: line 17
at EventTest.NoOneIsInterestedInThisEvent() in EventTest.cs: line 46

Then I updated the Execute() method in Eventful class:

public void Execute()
{
    // logic to get ready
 
    if (ContinuationHandler != null)
    {
        var args = new EventArgs();
        //stuff args with some data
        ContinuationHandler(this, args);
    }
}

This made all the 3 tests go Green!

Time and again, I find developers (including myself) over-relying on our automated tests. esp. unit tests which run fast and  reliably.

In the urge to save time today, we want to automate everything (which is good), but then we become blindfolded to this desire. Only later to realize that we’ve spent a lot more time debugging issues that our automated tests did not catch (leave the embarrassment caused because of this aside.)

I’ve come to believe:

A little bit of manual, sanity testing by the developer before checking in code can save hours of debugging and embarrassment.

Again this is contextual and needs personal judgement based on the nature of the change one makes to the code.

In addition to quickly doing some manual, sanity test on your machine before checking, it can be extremely important to do some exploratory testing as well. However, not always we can test things locally. In those cases, we can test them on a staging environment or on a live server. But its important for you to discover the issue much before your users encounter it.

P.S: Recently we faced Error CS0234: The type or namespace name ‘Specialized’ does not exist in the namespace ‘System.Collections’ issue, which prompted me to write this blog post.

Building on top of my previous blog entry: Version Control Branching (extensively) Considered Harmful

I always discourage teams from preemptively branching a release candidate and then splitting their team to harden the release while rest of the team continues working on next release features.

My reasoning:

• Increases the work-in-progress and creates a lot of planning, management, version-control, testing, etc. overheads.
• In the grand scheme of things, we are focusing on resource utilization, but the throughput of the overall system is actually reducing.
• During development, teams get very focused on churning out features. Subconsciously they know there will be a hardening/optimization phase at the end, so they tend to cut corners for short-term speed gains. This attitude had a snowball effect. Overall encourages a “not-my-problem” attitude towards quality, performance and overall usability.
• The team (developers, testers and managers) responsible for hardening the release have to work extremely hard, under high pressure causing them to burn-out (and possibly introducing more problems into the system.) They have to suffer for the mistakes others have done. Does not seem like a fair system.
• Because the team is under high pressure to deliver the release, even though they know something really needs to be redesigned/refactored, they just patch it up. Constantly doing this, really creates a big ball of complex mud that only a few people understand.
• Creates a “Knowledge/Skill divide” between the developers and testers of the team. Generally the best (most trusted and knowledgable) members are pick up to work on the release hardening and performance optimization. They learn many interesting things while doing this. This newly acquired knowledge does not effectively get communicate back to other team members (mostly developers). Others continue doing what they used to do (potentially wrong things which the hardening team has to fix later.)
• As releases pass by, there are fewer and fewer people who understand the overall system and only they are able to effectively harden the project. This is a huge project risk.
• Over a period of time, every new release needs more hardening time due to the points highlighted above. This approach does not seem like a good strategy of getting out of the problem.

If something hurts, do it all the time to reduce the pain and get better at it.

Hence we should build release hardening as much as possible into the routine everyday work. If you still need hardening at the end, then instead of splitting the teams, we should let the whole swamp on making the release.

Also usually I notice that if only a subset of the team can effectively do the hardening, then its a good indication that the team is over-staffed and that might be one of the reasons for many problems in the first place. It might be worth considering down-sizing your team to see if some of those problems can be addressed.

When confronted with Legacy code, we usually run into the Test-Refactor dilemma. To refactor code we need tests, to write tests, we need to refactor the code.

Some people might advise you to invest time upfront to create a whole set of tests. Instead I recommend that every time you touch a piece of legacy code (either to fix a bug or to enhance the functionality), you perform a couple of safe refactoring to enable you to create a few scaffolding tests, then clean up the code (may be even test drive the new code) and then get rid of the scaffolding tests.

Even though this approach might appear to be slower, why does this approach work better?

• You start seeing some immediate returns.
• On any given system, there are parts of the system which are more fragile and needs more attention than others. There are parts of code which we actively touch and others we rarely touch. When we have a limit time, it does not make sense in investing effort to create tests for areas that are fairly stable or rarely changed. Big upfront test creation might not take this aspect into account. So you might not get the biggest bang for your buck.
• The first few tests we usually write are fragile tests. But we won’t get this feedback nor the opportunity to improve the quality of our tests until its too late.
• When we get into a test creation mode, everyone is focusing on creating more and more tests. Finally when we start using the tests we’ve created, a small change in production code might breaks a whole bunch of tests. First few times developers wonder what happened, but if this generates a lot of false-negative (which usually they do), then developers start ignoring or deleting those tests. So the investment does not really pay for itself.
• Also when we have a whole lot of tests prematurely written, they start getting in the way of refactoring and genuinely improving the design of the code. (Defeats the whole point of creating test upfront so we can refactor.)
• People get too attached to the tests they had written (it was a big investment). They somehow want to make the test work. People fail to realize that those fragile tests are slowing them down and getting in their way.
• Unless the team gets into the habit of gradually building better test coverage, they will always play the catch up game with requirements constantly changing. (Remember we are chasing a moving target.)
• Its usually hard to take a fixed (usually long) duration of time off from CRs and bug fixes. People will be forced to multi-task.

I encourage you to share your own experience.

Technical Debt is any technical issues slowing down the project due to hasty (short-sighted) decisions made at an earlier point.

All of us make bad decisions, but not fixing them and just differing them really leads to bigger problems as these issues have a snowball effect.

Technical debt can be introduced at various levels:

• Code smells is the most obvious one,
• But things like lack of (or poor) automation,
• poor choice of tools,
• fragility in the development environment
• and so on

can also contribute to technical debt.

Most software organizations today suffer from what I called the “Inverted Testing Pyramid”. They spend maximum time and effort building end-to-end GUI test. Very little effort is spent on building unit/micro tests. Hence they end up with majority (80-90%) of their tests being end-to-end GUI tests. Some effort is spent on writing so-called “Integration test” (typically 5-15%.) Resulting in a shocking 1-5% of their tests being unit/micro tests.

Why is this a problem?

• The base of the pyramid is constructed from end-to-end GUI test, which are famous for their fragility and complexity. A small pixel change in the location of a UI component can result in test failure. GUI tests are also very time-sensitive, sometimes resulting in random failure (false-negative.)
• To make matters worst, most teams struggle automating their end-to-end tests early on, which results in huge amount of time spent in manual regression testing. Its quite common to find test teams struggling to catch up with development. This lag causes many other hard-development problems.
• Number of end-to-end tests required to get a good coverage is much higher and more complex than the number of unit tests + selected end-to-end tests required. (BEWARE: Don’t be Seduced by Code Coverage Numbers)

• Maintain a large number of end-to-end tests is quite a nightmare for teams. Following are some core issues with end-to-end tests:
  • It requires deep domain knowledge and high technical skills to write quality end-to-end tests.
  • They take a lot of time to execute.
  • They are relatively resource intensive.
  • Testing negative paths in end-to-end tests is very difficult (or impossible) compared to lower level tests.
  • When an end-to-end test fails, we don’t get pin-pointed feedback about what went wrong.
  • They are more tightly coupled with the environment and have external dependencies, hence fragile. Slight changes to the environment can cause the tests to fail. (false-negative.)
  • From a refactoring point of view, they don’t give the same comfort feeling to developers as unit tests can give.

Again don’t get me wrong. I’m not suggesting end-to-end integration tests are a scam. I certainly think they have a place and time.

What I propose and help many organizations achieve is the right balance of end-to-end tests, acceptance tests and unit tests. I call this “Inverting the Testing Pyramid.” [Inspired by Jonathan Wilson's book called Inverting The Pyramid: The History Of Football Tactics].

In a later blog post I can quickly highlight various tactics used to invert the pyramid.