“Release Early, Release Often” is a proven mantra, but what happens when you push this practice to it’s limits? .i.e. deploying latest code changes to the production servers every time a developer checks-in code?

At Industrial Logic, developers are deploying code dozens of times a day, rapidly responding to their customers and reducing their “code inventory”.

This talk will demonstrate our approach, deployment architecture, tools and culture needed for CD and how at Industrial Logic, we gradually got there.

Process/Mechanics

This will be a 60 mins interactive talk with a demo. Also has a small group activity as an icebreaker.

Key takeaway: When we started about 2 years ago, it felt like it was a huge step to achieve CD. Almost a all or nothing. Over the next 6 months we were able to break down the problem and achieve CD in baby steps. I think that approach we took to CD is a key take away from this session.

Talk Outline

1. Context Setting: Need for Continuous Integration (3 mins)
2. Next steps to CI (2 mins)
3. Intro to Continuous Deployment (5 mins)
4. Demo of CD at Freeset (for Content Delivery on Web) (10 mins) – a quick, live walk thru of how the deployment and servers are set up
5. Benefits of CD (5 mins)
6. Demo of CD for Industrial Logic’s eLearning (15 mins) – a detailed walk thru of our evolution and live demo of the steps that take place during our CD process
7. Zero Downtime deployment (10 mins)
8. CD’s Impact on Team Culture (5 mins)
9. Q&A (5 mins)

Target Audience

• CTO
• Architect
• Tech Lead
• Developers
• Operations

Context

Industrial Logic’s eLearning context? number of changes, developers, customers , etc…?

Industrial Logic’s eLearning has rich multi-media interactive content delivered over the web. Our eLearning modules (called Albums) has pictures & text, videos, quizes, programming exercises (labs) in 5 different programming languages, packing system to validate & produce the labs, plugins for different IDEs on different platforms to record programming sessions, analysis engine to score student’s lab work in different languages, commenting system, reporting system to generate different kind of student reports, etc.

We have 2 kinds of changes, eLearning platform changes (requires updating code or configuration) or content changes (either code or any other multi-media changes.) This is managed by 5 distributed contributors.

On an average we’ve seen about 12 check-ins per day.

Our customers are developers, managers and L&D teams from companies like Google, GE Energy, HP, EMC, Philips, and many other fortune 100 companies. Our customers have very high expectations from our side. We have to demonstrate what we preach.

Learning outcomes

• General Architectural considerations for CD
• Tools and Cultural change required to embrace CD
• How to achieve Zero-downtime deploys (including databases)
• How to slice work (stories) such that something is deployable and usable very early on
• How to build different visibility levels such that new/experimental features are only visible to subset of users
• What Delivery tests do
• You should walk away with some good ideas of how your company can practice CD

Slides from Previous Talks

Today I was adding some new users to our Hudson CI Server. Since we use Husdon’s Own User Database, I had to add new users in Hudson. However after that, when the new user tried to login, the user kept getting the following error:

Access denied User is missing the Read permission.

Google did not reveal any thing obvious. Then I started looking at Hudson configuration and realized that we’re using Hudson’s Matrix-based security authorization.

So every time we add a new user, we have to add the same user here as well. What a pain! Ideally permissions should be a link right from the new user’s page.

I wish we stop using the Martix-based security authorization and instead just use “Anyone can do anything” option. One less administrative step.

Currently at Industrial Logic we use Nginx as a reverse proxy to our Tomcat web server cluster.

Today, while running a particular report with large dataset, we started getting timeouts errors. When we looked at the Nginx error.log, we found the following error:

[error] 26649#0: *9155803 upstream timed out (110: Connection timed out) 
while reading response header from upstream, 
client: xxx.xxx.xxx.xxx, server: elearning.industriallogic.com, request: 
"GET our_url HTTP/1.1", upstream: "internal_server_url", 
host: "elearning.industriallogic.com", referrer: "requested_url"

After digging around for a while, I discovered that our web server is taking more than 60 secs to respond. Nginx has a directive called proxy_read_timeout which defaults to 60 secs. It determines how long nginx will wait to get the response to a request.

In nginx.conf file, setting proxy_read_timeout to 120 secs solved our problem.

server {
    listen       80;
    server_name  elearning.industriallogic.com;
    server_name_in_redirect off;
    port_in_redirect        off;
 
    location / {
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Real-Host  $host;
        proxy_read_timeout 120;
        ...
    }
    ...
}

Time and again, I find developers (including myself) over-relying on our automated tests. esp. unit tests which run fast and  reliably.

In the urge to save time today, we want to automate everything (which is good), but then we become blindfolded to this desire. Only later to realize that we’ve spent a lot more time debugging issues that our automated tests did not catch (leave the embarrassment caused because of this aside.)

I’ve come to believe:

A little bit of manual, sanity testing by the developer before checking in code can save hours of debugging and embarrassment.

Again this is contextual and needs personal judgement based on the nature of the change one makes to the code.

In addition to quickly doing some manual, sanity test on your machine before checking, it can be extremely important to do some exploratory testing as well. However, not always we can test things locally. In those cases, we can test them on a staging environment or on a live server. But its important for you to discover the issue much before your users encounter it.

P.S: Recently we faced Error CS0234: The type or namespace name ‘Specialized’ does not exist in the namespace ‘System.Collections’ issue, which prompted me to write this blog post.

Building on top of my previous blog entry: Version Control Branching (extensively) Considered Harmful

I always discourage teams from preemptively branching a release candidate and then splitting their team to harden the release while rest of the team continues working on next release features.

My reasoning:

• Increases the work-in-progress and creates a lot of planning, management, version-control, testing, etc. overheads.
• In the grand scheme of things, we are focusing on resource utilization, but the throughput of the overall system is actually reducing.
• During development, teams get very focused on churning out features. Subconsciously they know there will be a hardening/optimization phase at the end, so they tend to cut corners for short-term speed gains. This attitude had a snowball effect. Overall encourages a “not-my-problem” attitude towards quality, performance and overall usability.
• The team (developers, testers and managers) responsible for hardening the release have to work extremely hard, under high pressure causing them to burn-out (and possibly introducing more problems into the system.) They have to suffer for the mistakes others have done. Does not seem like a fair system.
• Because the team is under high pressure to deliver the release, even though they know something really needs to be redesigned/refactored, they just patch it up. Constantly doing this, really creates a big ball of complex mud that only a few people understand.
• Creates a “Knowledge/Skill divide” between the developers and testers of the team. Generally the best (most trusted and knowledgable) members are pick up to work on the release hardening and performance optimization. They learn many interesting things while doing this. This newly acquired knowledge does not effectively get communicate back to other team members (mostly developers). Others continue doing what they used to do (potentially wrong things which the hardening team has to fix later.)
• As releases pass by, there are fewer and fewer people who understand the overall system and only they are able to effectively harden the project. This is a huge project risk.
• Over a period of time, every new release needs more hardening time due to the points highlighted above. This approach does not seem like a good strategy of getting out of the problem.

If something hurts, do it all the time to reduce the pain and get better at it.

Hence we should build release hardening as much as possible into the routine everyday work. If you still need hardening at the end, then instead of splitting the teams, we should let the whole swamp on making the release.

Also usually I notice that if only a subset of the team can effectively do the hardening, then its a good indication that the team is over-staffed and that might be one of the reasons for many problems in the first place. It might be worth considering down-sizing your team to see if some of those problems can be addressed.

Branching is a powerful tool in managing development and releases using a version control system. A branch produces a split in the code stream.

One recurring debate over the years is what goes on the trunk (main baseline) and what goes on the branch. How do you decide what work will happen on the trunk, and what will happen on the branch?

Here is my rule:

Branch rarely and branch as late as possible. .i.e. Branch only when its absolutely necessary.

You should branch whenever you cannot pursue and record two development efforts in one branch. The purpose of branching is to isolate development effort. Using a branch is always more involved than using the trunk, so the trunk should be used in majority of the cases, while the branch should be used in rare cases.

According to Ned, historically there were two branch types which solved most development problems:

• Fixes Branch: while feature work continues on the trunk, a fixes branch is created to hold the fixes to the latest shipped version of the software. This allows you to fix problems without having to wait for the latest crop of features to be finished and stabilized.
• Feature Branch: if a particular feature is disruptive enough or speculative enough that you don’t want the entire development team to have to suffer through its early stages, you can create a branch on which to do the work.

Continuous Delivery (and deployment) goes one step further and solves hard development problems and in the process avoiding the need for branching.

What are the problems with branching?

• Effort, Time and Cognitive overhead taken to merge code can be actually significant. The later we merge the worse it gets.
• Potential amount of rework and bugs introduced during the merging process is high
• Cannot dynamically create a CI build for each branch, hence lack of feedback
• Instead of making small, safe and testable changes to the code base, it encourages developers to make big bang, unsafe changes

P.S: I this blog, I’m mostly focusing on Client-Server (centralized) version control system. Distributed VCS (DVCS) have a very different working model.

Impact of Continuous Integration on Team Culture:

• Encourages an Evolutionary Design and Continuous Improvement culture
• On complex projects, forces a nicely decoupled design such that each modules can be independently tested. Also ensures that in production you can support different versions of each module.
• Team takes shared ownership of their development and build process
• The source control trunk is in an always-working-state (avoid multiple branch issues)
  • No developer is blocked because they can’t get stable code
• Developers break down work into small end-to-end, testable slices and checks-in multiple times a day
  • Developers are up-to date with other developer changes
  • Team catches issues at the source and avoids last minute integration nightmares
  • Developers get rapid feedback once they check-in their code
    • Builds are optimized and parallelized for speed
      • Builds are incremental in nature (not big bang over-night builds)
    • Builds run all the automated tests (may be staged) to give realistic feedback
      • Captures and visualizes build results and logs very effectively
    • Display various source code quality metrics trends
      • Code coverage, cyclomatic complexity, coding convention violation, version control activity, bug counts, etc.
• Influence the right behavior in the team by acting as Information Radiator in the team area
  • Provide clear visual feedback about the build status
• Developers ask for an easy way to run and debug builds locally (or remotely)
• Broken builds are rare. However broken builds are rapidly fixed by developers
  • Build results are intelligently archived
  • Easy navigation between various build versions
    • Easily visualization and comparison of the change sets
• Large monolithic builds are broken into smaller, self contained builds with a clear build promotion process
• Complete traceability exists
  • Version Control, Project & Requirements Managements tool, Bug Tracking and Build system are completely integrated.
• CI page becomes the project dashboard for everyone (devs, testers, managers, etc.).

Any other impact you think is worth highlighting?

How good are you at limiting red time? .i.e. apply limiting WIP (Work-In-Progress) concept to Programming and Product Development.

What is Red Time?

• During Test Driven Development and Refactoring, time taken to fix compilation errors and/or failing tests.
• While Programming, time taken to get the logic right for a sub-set of the problem.
• While Deploying, downtime experienced by users
• While Integrating, time spent fixing broken builds
• While Planning and Designing, time spent before the user can use the first mini-version of the product
• And so on…

Basically time spent outside the safe, manageable state.

Let it be planning, programming or deploying, a growing group of practitioners have learned how to effectively reduce red time.

For example, there are many:

• Refactoring Strategies which can help you reduce your red time by keeping you in a state where you can take really safe steps to ensure the tests are always running.
• Zero-Downtime Deployment which helps you deploy new versions of the product without your customers experiencing any downtime.
• Continuous Deployment which helps you get a change made to code straight to your customers as efficiently as possible
• Lean Start-up techniques which helps validate business hypothesis in a safe, rapid and lean manner.
• And so on…

I highly recommend watching Joshua Kerievsky’s video on Limited Red Society to gain his insights.

Over the years we’ve realized that it always helps to have simple tools to visualize your red time. Visualization helps you understand what’s happening better. And that helps in proactively finding ways to minimize red time.

At Industrial Logic we have a new product called Sessions which helps you visualize your programming session. It highlights your red time.

You’ve heard about limiting WIP (Work-In-Progress) but how good are you at limiting red time? Red time is when you have compilation errors and/or failing tests. A growing group of practitioners have learned how to effectively reduce red time while test-driving and refactoring code. To understand how to limit red time, it helps to visualize it.

In this talk, I demonstrated various strategies to limit your time in Red. We also analyzed a live programming sessions using graphs that clearly visualize red time. Participants learned what development processes help or hurt our ability to limit red time and gained an appreciation for the visual cues that can help make you a better developers and fellow member of the Limited Red Society.

Slides from the Presentation: