“Release Early, Release Often” is a proven mantra, but what happens when you push this practice to it’s limits? .i.e. deploying latest code changes to the production servers every time a developer checks-in code?

At Industrial Logic, developers are deploying code dozens of times a day, rapidly responding to their customers and reducing their “code inventory”.

This talk will demonstrate our approach, deployment architecture, tools and culture needed for CD and how at Industrial Logic, we gradually got there.

Process/Mechanics

This will be a 60 mins interactive talk with a demo. Also has a small group activity as an icebreaker.

Key takeaway: When we started about 2 years ago, it felt like it was a huge step to achieve CD. Almost a all or nothing. Over the next 6 months we were able to break down the problem and achieve CD in baby steps. I think that approach we took to CD is a key take away from this session.

Talk Outline

1. Context Setting: Need for Continuous Integration (3 mins)
2. Next steps to CI (2 mins)
3. Intro to Continuous Deployment (5 mins)
4. Demo of CD at Freeset (for Content Delivery on Web) (10 mins) – a quick, live walk thru of how the deployment and servers are set up
5. Benefits of CD (5 mins)
6. Demo of CD for Industrial Logic’s eLearning (15 mins) – a detailed walk thru of our evolution and live demo of the steps that take place during our CD process
7. Zero Downtime deployment (10 mins)
8. CD’s Impact on Team Culture (5 mins)
9. Q&A (5 mins)

Target Audience

• CTO
• Architect
• Tech Lead
• Developers
• Operations

Context

Industrial Logic’s eLearning context? number of changes, developers, customers , etc…?

Industrial Logic’s eLearning has rich multi-media interactive content delivered over the web. Our eLearning modules (called Albums) has pictures & text, videos, quizes, programming exercises (labs) in 5 different programming languages, packing system to validate & produce the labs, plugins for different IDEs on different platforms to record programming sessions, analysis engine to score student’s lab work in different languages, commenting system, reporting system to generate different kind of student reports, etc.

We have 2 kinds of changes, eLearning platform changes (requires updating code or configuration) or content changes (either code or any other multi-media changes.) This is managed by 5 distributed contributors.

On an average we’ve seen about 12 check-ins per day.

Our customers are developers, managers and L&D teams from companies like Google, GE Energy, HP, EMC, Philips, and many other fortune 100 companies. Our customers have very high expectations from our side. We have to demonstrate what we preach.

Learning outcomes

• General Architectural considerations for CD
• Tools and Cultural change required to embrace CD
• How to achieve Zero-downtime deploys (including databases)
• How to slice work (stories) such that something is deployable and usable very early on
• How to build different visibility levels such that new/experimental features are only visible to subset of users
• What Delivery tests do
• You should walk away with some good ideas of how your company can practice CD

Slides from Previous Talks

We have a few Red Hat Enterprise Linux servers, all run ConfigServer and Security (CSF), which is a Stateful Packet Inspection (SPI) firewall, Login/Intrusion Detection and Security application for Linux servers. Amongst various other things, it looks for port scans, multiple login failures and other things that it thinks are ominous, and locks out the originating IP address by rewriting the iptables firewall rules.

For example, if you try to connect to the same server via http, https, ssh and svn within some short window of time, you are quite likely to incur its wrath. Developers at Industrial Logic often lock themselves out by getting blacklisted.

Generally when this happens, we ssh into one of our other server, connect to the server that has blacklisted us, and execute the following command to see what is going on:

$ sudo /usr/sbin/csf -t

As you can see, csf blacklisted my IP for port scanning.

If your IP is the only record, you can flush the whole temporary block list by executing:

$ sudo /usr/sbin/csf -tf
DROP all opt — in !lo out * 117.193.150.62 -> 0.0.0.0/0
csf: 117.193.150.62 temporary block removed
csf: There are no temporary IP allows

Alternatively you can execute the following command to just remove a specific IP:

$ sudo /usr/sbin/csf -tr

The easiest way to find your (external) IP address is to visit http://www.whatsmyip.org/

If you have a static IP, then you can whitelist yourself by:

$ sudo /usr/sbin/csf -a

Today I was adding some new users to our Hudson CI Server. Since we use Husdon’s Own User Database, I had to add new users in Hudson. However after that, when the new user tried to login, the user kept getting the following error:

Access denied User is missing the Read permission.

Google did not reveal any thing obvious. Then I started looking at Hudson configuration and realized that we’re using Hudson’s Matrix-based security authorization.

So every time we add a new user, we have to add the same user here as well. What a pain! Ideally permissions should be a link right from the new user’s page.

I wish we stop using the Martix-based security authorization and instead just use “Anyone can do anything” option. One less administrative step.

Industrial Logic’s eLearning has a feature where students can upload their programming exercise and get automated, personalized feedback. We do various server-side analysis (automated critique) of the student’s code to score them and to give them feedback about how well they performed in their programming exercises. To do this we need to compile their code on our server.

Recently we upgraded our .Net Compiler from version 2.0 to version 3.5. In version 2.0 we had to provide a reference to System.dll file (located in c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\System.dll) for compiling. In version 3.5, they don’t have System.dll file. It was replaced by System.core.dll (located in c:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll)

After making this change, when we ran the compiler using c:\WINDOWS\Microsoft.NET\Framework\v3.5\Csc.exe we got the following error:

Microsoft (R) Visual C# 2008 Compiler version 3.5.30729.1
for Microsoft (R) .NET Framework version 3.5
Copyright (C) Microsoft Corporation. All rights reserved.

SomeCSharpClass.cs: error CS0234: The type or namespace name ‘Specialized’ does not exist in the namespace ‘System.Collections’ (are you missing an assembly reference?)

It turns out that ‘System.collections.specialized’ namespace used to exist in System.dll in .net 2.0. In 3.5 System.Core.dll (which replaced the System.dll) does not contain it. Hence the compile time error.

We’ve fixed this issue by adding both System.Core.dll (v3.5) and System.dll (v2.0) to the compiler reference path. Not sure if this is the right thing to do. But it seems to work.

As an Eclipse user, who still works with CVS for some projects, when I check-out a repository through Eclipse, I have the choice between

• extssh method: Enter the password only once,
• ext method: Enter the password at every operation, several times.

‘extssh’ is the obvious choice. And life is good, until you need to use the command line CVS client.

The latest version of CVS 1.12.13 from ftp.gnu.org does not support extssh method.

If you run a CVS command from the command-line on a project created by Eclipse, it gives the following error message:

$cvs log
cvs log: Unknown method (`extssh’) in CVSROOT.
cvs log: in directory .:
cvs log: ignoring CVS/Root because it does not contain a valid root.
cvs log: No CVSROOT specified! Please use the `-d’ option
cvs [log aborted]: or set the CVSROOT environment variable.

Unfortunately even after having reported this issue several years ago, the official cvs program does not support this method.

Luckily Eclipse offers an improved interoperability. You can specify ‘ext’ method while checking out the repository. Then via
Eclipse > Preferences > Team > CVS > Ext Connection Method

Select ‘Use another connection method type to connect’ and specify ‘Connection Type’ as ‘extssh’

If you had already created a project using extssh, don’t worry. You can always go the CVS Perspective > Select your repository > right click > Properties > change the connection type back to ext.

It will take a while for Eclipse to update the meta files on the disk. Once its done, you should be able to execute cvs commands from both Eclipse and the command line client.

What tools do you use for Code Analysis of C/C++ projects?

This is a common questions a lot of teams have when we discuss Continuous Integration in C/C++.

I would recommend the following tools:

UPDATE: I strongly recommend looking at CppDepend (commercial), one stop solution for all kinds of metric. It has some very cool/useful features like Code Query Language, Customer Build Reporting, Comparing Builds, great visualization diagrams for dependency, treemaps, etc.

• Cyclomatic Complexity
  • Cyclo – Open Source
  • CCCC – Open Source
  • McCabe IQ – Commercial
  • CMetrics – Open Source
  • Cyclomatic Complexity Viewer – Visual Studio .NET Addins
• Code Coverage
  • GCov – Open Source
  • Bullseye Coverage – Commercial
• LOC Metric
  • Most of the tools listed above would give you this measure
  • LC2 – Open Source
  • CNCC – Open Source
• Copy Paste Detector – PMD – Open Source

Wikipedia page on Static Code Analysis Tools has a list of many more tools.

This post is moved to Industrial Logic’s Blogic.

For many years now, our Agile eLearning albums have been offered at a flat, USD price, regardless of where you lived in the world.

That is about to change!

Our Agile eLearning albums are now available to individuals around the world at new, lower, regionally adjusted prices.

What Does This Mean To You?

It means that we have introduced a new Individual pricing model to complement our Corporate pricing model. The new model, based on Purchasing Power Parity (PPP), makes our Agile eLearning more affordable for individual purchasers.Individuals will still see USD album prices — however, they will be adjusted to reflect the purchasing power of your country’s currency.

As of April, 2010, if you live in India , you can purchase an Individual License to our Code Smells album in Java for USD $126.00, rather than the Corporate price of USD $210.00.

Even if you live in a country where the PPP discount is modest, Individual Licenses will still receive a 15% discount:

As of April, 2010, if you live in the United State , you can purchase an Individual License to our Code Smells album in Java for USD $179.00, rather than the Corporate price of USD $210.00.

Why Are We Doing This?

We’re devoted to helping people throughout the world acquire Agile skills at a reasonable price. We’ve helped large corporations, like Google, HP and Standard Life, realize this goal and now we’d like to make the same training affordable for individuals as well.To learn more about Agile eLearning, please visit our Greatest Hits Shop.

I was just looking through the Google Analytics on my blog, I was surprised to see Firefox dominate the browser usage race. Also surprised to see how fast Google Chrome has penetrated the market. (Over the years, Firefox has got really fat and slow. Chrome’s launch timing was perfect. Just when many geeks could not take FF’s speed and instability anymore.)

Yesterday, my blog had 3,697 visits from 16 different browsers.

And if you see the break up by Operating Systems

This data matches that from w3schools’ Browser Stats, except that my blog is not very IE friendly hence less IE traffic. Also I hope Windows usage goes down…

For Freeset, I’ve always been in the quest of Simplest Thing that Could Possibly Work. In a previous post, I explained how we’ve embraced an ultra-light process (call it lean, if you like) to build their e-commerce site.

In that post, I’ve talked about our wish to create a Selenium test suite for regression testing. But it never got high enough on our priority list. (esp. coz we mostly have static content served from a CMS as of now).

While that is something I wanted to tackle, last night, when I was moving Industrial Logic and Industrial XP‘s site over to a new server hardware, I wanted some quick way to test if all the pages were correctly displayed after the move. This was important since we switched from Apache to Nginx. Nginx has slightly different way to handle secure pages, etc.

So I asked on Twitter, if anyone knew of a tool that could compare 2 deployments of the same website. Few people responding saying I could use curl/wget with diff recursively. That seemed like the simplest thing that could work for now. So this morning I wrote a script.

rm -Rf * && mkdir live && cd live && wget -rkp -l5 -q -np -nH http://freesetglobal.com && cd .. && mkdir dev && cd dev && wget -rkp -l5 -q -np -nH http://dev.freesetglobal.com && cd .. && for i in `grep -l dev.freesetglobal.com \`find ./dev -name '*'\`` ; do sed -e 's/dev.freesetglobal.com/freesetglobal.com/g' $i > $i.xx && mv $i.xx $i; done && diff -r -y --suppress-common-lines -w -I '^.*' dev live

I’m planning to use this script to do simple regression test of our Freeset site. We have a live and a dev environment. We make changes on dev and frequently sync it up with live. I’m thinking before we sync up, we can check if we’ve made the correct changes to the intended pages. If some other pages show up in this diff that we did not expect, it’s a good way to catch such issue before the sync.

Note: One could also use diff with -q option, if all they are interested to know is which pages changes. Also note that under Mac, the sed command’s -i (inline edit) option is broken. It simply does not work as explained. If you give sed -i -e …., it ends up creating backup files with -e extension. #fail.