Many friends responded to my previous post on How Much Should You Think about the Long-Term? saying:

Even if the future is uncertain and we know it will change, we should always plan for the long-term. Without a plan, we cease to move forward.

I’m not necessarily in favor or against this philosophy. However I’m concerned about the following points:

• Effort: The effort required to put together an initial direction is very different from the effort required to put a (proper) plan together. Is that extra effort really worth it esp. when we know things will change?
• Attachment: Sometimes we get attached with our plans. Even when we see things are not quite inline with our plan, we think, its because we’ve not given enough time or we’ve not done justice to it.
• Conditioned: Sometimes I notice that when we have a plan in place, knowingly or unknowingly we stop watching out for certain things. Mentally we are at ease and we build a shield around us. We get in the groove of our plan and miss some wonderful opportunities along the way.

The amount of planning required seems to be directly proportional to the size of your team.

If your team consists of a couple of people, you can go fairly lightweight. And that’s my long-term plan to deal with uncertainty.

Often people tell you that “You should think about the long-term.”

Sometimes people tell you, “Forget long-term, its too vague, but you should at least think beyond the near-term.”

Really?

Unfortunately, part of my brain (prefrontal cortex), which can see and analyze the future, has failed to develop compared to the other smart beings.

At times, I try to fool myself saying I can anticipate the future, but usually when I get there (future) its quite different. I realize that the way I think about the future is fundamentally flawed. I take the present and fill it with random guesses about something that might happen. But I always miss things that I’m not aware of or not exposed to.

In today’s world, when there are a lot of new ideas/stuff going around us, I’m amazed how others can project themselves into the future and plan their long-terms?

Imagine a tech-company planning their long-term plan, 5-years ago, when there were no iPads/tablets. They all must have guessed a tablet revolution and accounted that in their long-term plans. Or even if they did not, it would have been easy for them to embrace it right?

You could argue that the tablet revolution is a one-off phenomenon or an outlier. Generally things around here are very predictable and we can plan our long-term without an issue. Global economics, stability of government, rules and regulations, emergence of new technologies, new markets, movement of people, changes in their aspirations, environmental issues, none of these impact us in any way.

Good for you! Unfortunately I don’t live in a world like that (or at least don’t fool myself thinking about the world that way.)

By now, you must be aware that we live in a complex adaptive world and we humans ourselves are complex adaptive system. In complex adaptive system, the causality is retrospectively coherent. .i.e. hindsight does not lead to foresight.

10 years ago, when I first read about YAGNI and DTSTTCPW, I thought that was profound. It was against the common wisdom of how people designed software. Software development has come a long way since then. But is XP the best way to build software? Don’t know. Question is, people who used these principles did they build great systems? Answer is: Yes, quite a few of them.

In essence, I think one should certainly think about the future, make reasonable (quick) guesses and move on. Key is to always keep an open mind and listen to subtle changes around us.

One cannot solely rely on their “intuition” about long term. Arguing on things you’ve all only guessed seems like a huge waste of time and effort. Remember there is a point of diminishing returns and more time you spend thinking long-term, less your chances are of getting it right.

I’m a strong believer of “Action Precedes Clarity!”

Update: In response to many comments: When the Future is Uncertain, How Important is A Long-Term Plan?

Every single line of code must be unit tested!

This sound advice rather seems quite extreme to me. IMHO a skilled programmer pragmatically decides when to invest in unit testing.

After practicing (automated) unit testing for over a decade, I’m a strong believer and proponent of automated unit testing. My take on why developers should care about Unit Testing and TDD.

However over the years I’ve realized that automated unit tests do have four, very important, costs associated with them:

• Cost of writing the unit tests in the first place
• Cost of running the unit tests regularly to get feedback
• Cost of maintaining and updating the unit tests as and when required
• Cost of understanding other’s unit tests

I always knew and believed that I need to exercise and stay in shape to be effective at work. But for few years other burning issues always forced me away from exercising. I kept procrastinating!

Gradually, inch-by-inch, year after year, I found myself becoming a potato. It got to a point where it felt like there was no going back.

Unfortunately many professionals (including myself) do the exact same thing when it comes to Learning and staying up-to-date (keeping our skills in shape.)

We all know the importance of learning, yet we always find reasons not to invest right now.

But why?

If you Google for “avoid procrastination”, you’ll find tons of techniques to help you avoid procrastination. But I don’t know a single person who does not suffer from this.

Luckily for me, from the last 3 years, I’ve really been exercising every day (almost everyday.) On a given day, if I don’t, I feel really crappy.

So far I’ve not seen a similar behavior when it comes to Learning.

So what made me change my attitude towards exercising but not learning?

• Most important thing, with exercising, both the outcome and output is easy to measure/judge.
  • There are some very easy measurements that can quickly show you the difference between being in-shape or not.
  • Did I workout for 30 mins everyday and is my weight stable? (Output)
  • Do I feel energized and fresh the whole day? (Outcome) (I can easily judge this.)
  • But with learning, how do I measure the outcome? Its a lot more vague and much more long term.
• Few things changed, I was in a company of people (at work, back home, amongst friends), who were all very conscious about staying in shape. Being surrounded by right people, who felt much better after staying in shape, reinforced my thinking and encouraged me to take the first step. Unfortunately its hard to find similar environment when it comes to learning.
• I was able to set some simple targets (run for 30 mins every day) and measure if I met the targets or not. If I did not meet those targets, the results were quite obviously immediately. Unfortunately when it comes to Learning I’m not able to find a simple model like this, esp. results being immediately obvious.
• Personally for me, exercising, which involves mostly physical work, is relatively easy even when I’m tired or feeling lazy. But its hard to learn under such conditions because learning requires high cognitive processing.
• There are many options for exercising, but I was able to quickly decide on running everyday as its the simplest thing I could do anywhere and still get a good workout. But when in comes to learning, we have so many options and its intimidating to even get started.
• Exercising usually is fun. Its enjoyable. There is an immediate positive feedback cycle. With learning that’s not always the case. Many times its frustrating.

When it comes to learning, I’ve tried to set various targets like:

• Everyday for 30 mins, either read a book or blogs or twitter or practice solving some design problems.
  • Problem is, even after doing this, I’m not sure if I’m really benefiting or not. May be in the long run, but not now.
• Write a blog post every other day. Hoping that, to write a meaningful blog post, I would go read something at least to clarify my thinking.
  • Problem is, I continue to write crap without reading much and I don’t write that regularly either.

What have you tried to keep up with your learning?

P.S: I learn a lot at my work, but its primarily driven by the need at hand. Which is good. But the broader learning and staying up-to-date with things is equally important and that’s what I’m talking about.

Building on top of my previous blog entry: Version Control Branching (extensively) Considered Harmful

I always discourage teams from preemptively branching a release candidate and then splitting their team to harden the release while rest of the team continues working on next release features.

My reasoning:

• Increases the work-in-progress and creates a lot of planning, management, version-control, testing, etc. overheads.
• In the grand scheme of things, we are focusing on resource utilization, but the throughput of the overall system is actually reducing.
• During development, teams get very focused on churning out features. Subconsciously they know there will be a hardening/optimization phase at the end, so they tend to cut corners for short-term speed gains. This attitude had a snowball effect. Overall encourages a “not-my-problem” attitude towards quality, performance and overall usability.
• The team (developers, testers and managers) responsible for hardening the release have to work extremely hard, under high pressure causing them to burn-out (and possibly introducing more problems into the system.) They have to suffer for the mistakes others have done. Does not seem like a fair system.
• Because the team is under high pressure to deliver the release, even though they know something really needs to be redesigned/refactored, they just patch it up. Constantly doing this, really creates a big ball of complex mud that only a few people understand.
• Creates a “Knowledge/Skill divide” between the developers and testers of the team. Generally the best (most trusted and knowledgable) members are pick up to work on the release hardening and performance optimization. They learn many interesting things while doing this. This newly acquired knowledge does not effectively get communicate back to other team members (mostly developers). Others continue doing what they used to do (potentially wrong things which the hardening team has to fix later.)
• As releases pass by, there are fewer and fewer people who understand the overall system and only they are able to effectively harden the project. This is a huge project risk.
• Over a period of time, every new release needs more hardening time due to the points highlighted above. This approach does not seem like a good strategy of getting out of the problem.

If something hurts, do it all the time to reduce the pain and get better at it.

Hence we should build release hardening as much as possible into the routine everyday work. If you still need hardening at the end, then instead of splitting the teams, we should let the whole swamp on making the release.

Also usually I notice that if only a subset of the team can effectively do the hardening, then its a good indication that the team is over-staffed and that might be one of the reasons for many problems in the first place. It might be worth considering down-sizing your team to see if some of those problems can be addressed.

My friend Srihari has been working on a great project called Systems We Make. His goal is to provide a consolidated catalog of various papers/articles about building large, complex, distributed system.

He personally searches, reads through and filters each the papers for you. This is awesome!

Please support his initiative.

Most eLearning simply automate the formal class-room training environment. Only difference is students can be distributed now.

Although a lot of money is spent on creating engaging, interactive, multimedia courseware, one vital ingredient is missing: Other People.

Many companies adopt a blended approach to bring in the other people aspect. But even face?to?face events are often delivered in a fairly traditional way.

Meanwhile outside of the formal learning environment in online colleges (online courses, classroom, workshops, etc) individuals are learning in other ways – often without realizing it ? whether it be by reading, listening or watching or simply by talking to one another.

Only recently informal learning been recognized as important part of learning and that it needs to be supported within an organization.

This presentation explains how:

1. There are imperatives for continual learning
2. Learning is a process, not a series of events
3. Most learning occurs outside classrooms
4. The vast majority of learning is social
5. A lot of formal learning is ineffective
6. People learn better when they are in charge
7. There’s inherent inertia in formal approaches
8. Informal and social learning are cost-effective

Today we are in a knowledge economy and upgrading employee’s knowledge portfolio is the most important investment companies can make. But in this obsession for learning, our industry is spending a lot of money and is very heavily relying on formal learning.

Companies fail to realize 75% of actual learning comes from informal learning. Learning happens as the learner creatively adopts and adapts to ever changing circumstances. Its important to acknowledge that informal learning is crucial to learning how to do anything.

According to a study of Time-to-Performance done by Sally Anne Moore at Digital Equipment Corporation in the early 1990s, and repeated by universities, other corporations, and even the Department of Health and Human Services, following image graphically shows the disparity between formal learning and informal learning.

In “At the Water Cooler of Learning” article by David Grebow, he explains this graph with a very nice example.

IMHO formal training might be a good start, but deliberate practice on pet projects and on real world projects is the source of real learning. Necessity is not just the mother of innovation, but learning too.

To amplify your active informal-learning, make sure you:

• spend enough time at the water-coolers in your office
• attend user group meetings
• participate in online forums
• follow interesting people on twitter
• regularly read other people’s blogs (and source code)
• leverage all the Web 2.o Social Media forums
• attend a conference (esp. open space or workshop based conferences)
• and so on…

What is the purpose of Continuous Integration (CI)?

To avoid last minute integration surprises. CI tries to break the integration process into small, frequent steps to avoid big bang integration as it leads to integration nightmare.

If people are afraid to check-in frequently, your Continuous Integration process is not working.

CI process goes hand in hand with Collective Code Ownership and Single-Team attitude.

CI is the manifestation of “Stop the Line” culture from Lean Manufacturing.

What are the advantages of Continuous Integration?

• Helps to improve the quality of the software and reduce the risk by giving quicker feedback.
  • Experience shows that a huge number of bugs are introduced during the last-minute code integration under panic conditions.
• Brings the team together. Helps to build collaborative teams.
• Gives a level of confidence to checkin code more frequently that was once not there.
• Helps to maintain the latest version of the code base in always shippable state. (for testing, demo, or release purposes)
• Encourages lose coupling and evolutionary design.
• Increase visibility and acts as an information radiator for the team.
• By integrating frequently, it helps us avoid huge integration effort in the end.
• Helps you visualize various trends about your source code. Can be a great starting point to improve your development process.

Is Continuous Integration the same as Continuous build?

No, continuous build only checks if the code compiles and links correctly. Continuous Integration goes beyond just compiling.

• It executes a battery of unit and functional tests to verify that the latest version of the source code is still functional.
• It runs a collection of source code analysis tools to give you feedback about the Quality of the source code.
• It executes you packing script to make sure, the application can be packaged and installed.

Of course, both CI and CB should:

• track changes,
• archive and visualize build results and
• intelligently publish/notify the results to the team.

How do you differentiate between Frequent Versus Continuous Integration?

Continuous means:

• As soon as there is something new to build, its built automatically. You want to fail-fast and get this feedback as rapidly as possible.
• When it stops becoming an event (ceremony) and becomes a behavior (habit).

Merge a little at a time to avoid the big cost at full integration at the end of a project. The bottom line is fail-fast & quicker feedback.

Can Continuous Integration be manual?

Manual Continuous Integration is the practice of frequently integrating with other team members’ code manually on developer’s machine or an independent machine.

Because people are not good at being consistent and cannot do repetitive tasks (its a machine’s job), IMHO, this process should be automated so that you are compiling, testing, inspecting and responding to feedback.

What are the Pre-Requisites for Continuous Integration?

This is a grey area. Here a quick list is:

• Common source code repository
• Source Control Management tool
• Automated Build scripts
• Automated tests
• Feedback mechanism
• Commit code frequently
• Change of developer mentality, .i.e. desire to get rapid feedback and increase visibility.

What are the various steps in the Continuous Integration build?

• pulling the source from the SCM
• generating source (if you are using code generation)
• compiling source
• executing unit tests
• run static code analysis tools – project size, coding convention violation checker, dependency analysis, cyclomatic complexity, etc.
• generate version control usage trends
• generate documentation
• setup the environment (pre build)
• set up third party dependency. Example: run database migration scripts
• packaging
• deployment
• run various regression tests: smoke, integration, functional and performance test
• run dynamic code analysis tools – code coverage, dead-code analyzer,
• create and test installer
• restore the environment (post build)
• publishing build artifact
• report/publish status of the build
• update historical record of the build
• build metrics – timings
• gather auditing information (i.e. why, who)
• labeling the repository
• trigger dependent builds

Who are the stakeholders of the Continuous Integration build?

• Developers
• Testers [QA]
• Analysts/Subject Matter Experts
• Managers
• System Operations
• Architects
• DBAs
• UX Team
• Agile/CI Coach

What is the scope of QA?

They help the team with automating the functional tests. They pick up the product from the nightly build and do other types of testing.
For Ex: Exploratory testing, Mutation testing, Some System tests which are hard to automate.

What are the different types of builds that make Continuous Integration and what are they based on?

We break down the CI build into different builds depending on their scope & time of feedback cycle and the target audience.

1. Local Developer build :
1.a. Job: Retains the environment. Only compiles and tests locally changed code (incremental).
1.b. Feedback: less than 5 mins.
1.c. Stakeholders: Developer pair who runs the build
1.d. Frequency: Before checking in code
1.e. Where: On developer workstation/laptop

2. Smoke build :
2.a. Job: Compiles , Unit test , Automated acceptance and Smoke tests on a clean environment[including database].
2.b. Feedback: less than 10 to 15 mins. (If it takes longer, then you could make the build incremental, not start on a clean environment)
2.c. Stakeholders: All the developers within a single team.
2.d. Frequency: With every checkin
2.e. Where: On a team’s dedicated continuous integration server. [Multiple modules can share the server, if they have parallel builds]

3. Functional build :
3.a. Job: Compiles , Unit test , Automated acceptance and All Functional\Regression tests on a clean environment. Stubs/Mocks out other modules or systems.
3.b. Feedback: less than 1 hour.
3.c. Stakeholders: Developers , QA , Analysts in a given team
3.d. Frequency: Every 2 to 3 hours
3.e. Where: On a team’s dedicated continuous integration server.

4. Cross module build :
4.a. Job: If your project has multiple teams, each working on a separate module, this build integrates those modules and runs the functional build across all those modules.
4.b. Feedback: in less than 4 hr.
4.c. Stakeholders: Developers , QA , Architects , Manager , Analyst across the module team
4.d. Frequency: 2 to 3 times a day
4.e. Where: On a continuous integration server owned by all the modules. [Different from above]

5. Product build :
5.a. Job: Integrates all the code that is required to create a single product. Nothing is mocked or stubbed. [Except things that are not yet built]. Creates all the artifacts and publishes a deployable product.
5.b. Feedback: less than 10 hrs.
5.c. Stakeholders: Every one including the Project Management.
5.d. Frequency: Every night.
5.e. Where: On a continuous integration server owned by all the modules. [Same as above]

General Rule of Thumb: No silver bullet. Adapt your own process/practice.