Remember the dot-com days of Webvan and Pets.com? We took traditional businesses and gave then an online presence. Rapidly acquiring a large customer base was the sole goal of many dot-coms. “If we can get enough users, we can easily figure out how to monetize it.” And all of this made perfect sense expressed in dollars and cents. I know people who melted down Yahoo Finance’s servers by checking for their favourite stocks prices throughout the day, calculating their (paper) net worth in real time. If you were not part of this madness, you were certainly considered stupid.

But then on March 10, 2000, the perspective changed. Suddenly it became clear that this was really a bubble. Without having real profits (or even revenue/cash-flow), it was really just a house of cards. In hindsight, the entire dot-com burst made perfect sense. But why wasn’t this obvious to everyone (including me) to start with?

In complex adaptive system, the causality is retrospectively coherent. .i.e. hindsight does not lead to foresight. When we look back at the events, we can (relatively) easily construct a theory to explain the rationale behind the occurrence of these events. In fact, when we look back, the reasons are so obvious that one can easily be fooled into believing that “Only if we spend more time, carefully analysing and thinking through the situation at hand, we can completely avoid unwanted events in future.” Yet, time and again, we’ve always been caught by surprise and it almost appears to be impossible to predict such events ahead of time. Call it the Black Swan effect or whatever name you fancy.

This effect gives rise to a classic management dilemma – Predictability Paradox(pdf). In the zeal to improve the effectiveness and reliability of software development, managers institutionalise practices that unfortunately decrease, rather than increase, the predictability of the product’s success. Most companies spend an awful lot of effort and money to analyse the past, derive patterns and best practices, set targets and create processes to prevent past failure and produce ideal future goals. If software development was highly structured, if we had a stable environment and we had a good data points from million other projects, this approach might work. But for software development, which is a creative-problem solving domain, with high levels of uncertainty and each project having an unique context, these techniques (best practices) are rather dangerous.

In our domain,

• We need to break the vague problem down into small safe-fail experiments.
• Then execute each experiment in short iterative and incremental cycles.
• We need to focus on tight feedback loops, which will help us adapt & co-evolve the system. (We cannot be stuck with analysis paralysis.)
• We need to probe the system with experiments and find emergent practices.
• And then apply these practices in a given context, for a short duration.
• Speed and Sustainability are extremely important factors.

This is what I mean when I say “Action Precedes Clarity”.

There is an old saying that “Practice makes perfect“. But not all practice can make you perfect. There is a specific type of practice usually referred to as deliberate practice or deep practice which can really help you master the skill at hand.

Geoff Colvin, the author of the groundbreaking bestseller Talent Is Overrated explains that deliberate practice can be described by these five characteristics:

1. It’s designed specifically to improve performance
2. It can be repeated a lot
3. Feedback on actions is continuously available
4. It’s highly demanding mentally
5. It isn’t much fun

Hence a short, focused and regular practice session is lot more effective than one long, generic, random practice session.

Its also important to note that deliberate/deep practice requires a certain context or mindset to really facilitate the learning. Following are important characteristics:

• You are slightly off-balance, trying to get back
• You are constantly getting tons of clear, instant feedback
• You are at the edge of your ability, but motivated to stretch yourself a little bit more
• You are staring at you role models, .i.e. you’ve a clear idea of your goals or the kind of person you want to become.

The advantage of deliberate practice is cumulative, hence starting early has a big advantage.

Generally, I used to be against rote learning, but now I’m rethinking through my conclusions.

If this topic interests you, I would strongly recommend the following books:

• Talent Is Overrated by Geoff Colvin
• The Talent Code by Daniel Coyle

Some good videos to start on this topic:

The third workshop we are covering is ‘Problem-Solving and Decision-Making in Software Development‘ by Linda Rising.

Linda is well known for her book ‘Fearless change Patterns for Introducing New Idea‘. It is cited for providing proven techniques, formulated as patterns, for implementing change in teams and organizations. She is well respected for her work on patterns and their application in the workplace.

Linda has delivered keynotes at many agile conferences and is an internationally well-known speaker. We are really looking forward to her keynote at Agile India 2013 and also this particular workshop.

This week we spent some time with Linda to get further information on her workshop and to discuss her keen interest in how the brain works.

What motivated you to do research in cognitive science?

When Mary Lynn and I were writing Fearless Change, one of our reviewers said that the reason the patterns worked was because they were based on influence strategies. I had heard of “influence” but didn’t realize it was a special area of study in “social” psychology – the psychology of group behavior. I started looking at the growing research in that field and it led to my current passion to follow advances in cognitive psychology. This field is making enormous headway and it has significant impact on the way we work. We need to pay attention.

Can you give us an example of the application of cognitive science in software development?

The most important news from this area is the disturbing finding that we are not rational thinkers. I have believed this nearly all my very long life so it is difficult for me to accept, even as I read the pretty clear evidence. Even smart people are not rational. And, it’s not that we are not rational some of the time. The message is that we struggle to apply rational approaches to any of our decisions. This turns our development process upside down. In the past we believed that customers were rational, fellow developers and testers were rational and that was how we communicated and made decisions. Seeing that we make decisions based on what is not only not rational but not available to us or anyone else, means that we can’t make assumptions about what others say or do.

What is the take away for the attendees from the workshop?

It’s impossible to summarize all of the advances in cognitive science but I will try to hand attendees a list of practical tips that they can apply when they return to work. These tips will help them be better thinkers, better problem-solvers, and better able to influence others. I also hope to encourage all of the participants to become interested in this area. I can provide references to help them get started.

Who is the target audience for the workshop?

Anyone who works in any organization and must solve problems, make decisions, or convince others will benefit.

Some past talks by Linda: http://web.lindarising.info/Live_.html

Recently I facilitated a workshop at the Agile Goa 2012 Conference titled – “Agile Way of Dealing with Uncertainty in a Complex Adaptive World“.

Abstract: It is human nature to look for patterns while solving new problems. We have a dangerous tendency to reuse what we already know to solve the next problem. We rarely discard what we’ve learned; we simply build on top of it. Sometimes this is a useful tactic, but often new problems and their context are slightly (if not vastly) different than the previous ones. And applying our previous way of doing things, will not be best suited for tackling the new problem.

In the software world, we’ve seen a similar desire to find the “one true way”, “the BEST method”, “the silver bullet” to solve all software development problems. Alas, after decades of trying we’ve not found one.

In this workshop, I’ll let you discover why this is not possible and possibly explain how best to deal with this problem. This ideas in this workshop are based on my experience backed by latest research from Cognitive Science, Complex Adaptive System’s Theory and Evolutionary Psychology.

Slides:

Many friends responded to my previous post on How Much Should You Think about the Long-Term? saying:

Even if the future is uncertain and we know it will change, we should always plan for the long-term. Without a plan, we cease to move forward.

I’m not necessarily in favor or against this philosophy. However I’m concerned about the following points:

• Effort: The effort required to put together an initial direction is very different from the effort required to put a (proper) plan together. Is that extra effort really worth it esp. when we know things will change?
• Attachment: Sometimes we get attached with our plans. Even when we see things are not quite inline with our plan, we think, its because we’ve not given enough time or we’ve not done justice to it.
• Conditioned: Sometimes I notice that when we have a plan in place, knowingly or unknowingly we stop watching out for certain things. Mentally we are at ease and we build a shield around us. We get in the groove of our plan and miss some wonderful opportunities along the way.

The amount of planning required seems to be directly proportional to the size of your team.

If your team consists of a couple of people, you can go fairly lightweight. And that’s my long-term plan to deal with uncertainty.

Often people tell you that “You should think about the long-term.”

Sometimes people tell you, “Forget long-term, its too vague, but you should at least think beyond the near-term.”

Really?

Unfortunately, part of my brain (prefrontal cortex), which can see and analyze the future, has failed to develop compared to the other smart beings.

At times, I try to fool myself saying I can anticipate the future, but usually when I get there (future) its quite different. I realize that the way I think about the future is fundamentally flawed. I take the present and fill it with random guesses about something that might happen. But I always miss things that I’m not aware of or not exposed to.

In today’s world, when there are a lot of new ideas/stuff going around us, I’m amazed how others can project themselves into the future and plan their long-terms?

Imagine a tech-company planning their long-term plan, 5-years ago, when there were no iPads/tablets. They all must have guessed a tablet revolution and accounted that in their long-term plans. Or even if they did not, it would have been easy for them to embrace it right?

You could argue that the tablet revolution is a one-off phenomenon or an outlier. Generally things around here are very predictable and we can plan our long-term without an issue. Global economics, stability of government, rules and regulations, emergence of new technologies, new markets, movement of people, changes in their aspirations, environmental issues, none of these impact us in any way.

Good for you! Unfortunately I don’t live in a world like that (or at least don’t fool myself thinking about the world that way.)

By now, you must be aware that we live in a complex adaptive world and we humans ourselves are complex adaptive system. In complex adaptive system, the causality is retrospectively coherent. .i.e. hindsight does not lead to foresight.

10 years ago, when I first read about YAGNI and DTSTTCPW, I thought that was profound. It was against the common wisdom of how people designed software. Software development has come a long way since then. But is XP the best way to build software? Don’t know. Question is, people who used these principles did they build great systems? Answer is: Yes, quite a few of them.

In essence, I think one should certainly think about the future, make reasonable (quick) guesses and move on. Key is to always keep an open mind and listen to subtle changes around us.

One cannot solely rely on their “intuition” about long term. Arguing on things you’ve all only guessed seems like a huge waste of time and effort. Remember there is a point of diminishing returns and more time you spend thinking long-term, less your chances are of getting it right.

I’m a strong believer of “Action Precedes Clarity!”

Update: In response to many comments: When the Future is Uncertain, How Important is A Long-Term Plan?

I don’t think productivity is directly influenced by time alone. There are other important factors which make it or break it.

• There are days when I can get a lot done in lot less time and other days life moves slower than a snail.
• There are days when I can stay very focused and productive for up to 15-16 hrs and there are days when couple of hours of work seems like torture.
• There are certain type of tasks that I can get very productive very quickly and certain type of tasks which takes times to get into the flow.
• If I have a vested interest (financial, reputation, learning, etc.), I train myself to stay focused and productive longer (can back-fire as well.)
• There are phases of one’s life when one can clock in more hours and there are phases where one cannot; other things are higher priority. If forced (soft or hard), it usually leads to wrong behavior.
• I’ve seen many people who are very deadline driven (including myself.) I get a lot done when I’m working against hard deadlines. Yes, quality certainly takes a hit, but in many cases delivering slightly lower quality stuff is more important than not delivering at all or delivering later.

If we look at these various aspects, we get close to a start-up environment. And under these conditions, we do see teams being quite productive and delivering interesting products.

Applying this in a different context/environment usually back-fires.

What do you think?

This blog post was trigger by Tim Berry's blog post on Productivity Paradox: Maybe Less is More.

I always knew and believed that I need to exercise and stay in shape to be effective at work. But for few years other burning issues always forced me away from exercising. I kept procrastinating!

Gradually, inch-by-inch, year after year, I found myself becoming a potato. It got to a point where it felt like there was no going back.

Unfortunately many professionals (including myself) do the exact same thing when it comes to Learning and staying up-to-date (keeping our skills in shape.)

We all know the importance of learning, yet we always find reasons not to invest right now.

But why?

If you Google for “avoid procrastination”, you’ll find tons of techniques to help you avoid procrastination. But I don’t know a single person who does not suffer from this.

Luckily for me, from the last 3 years, I’ve really been exercising every day (almost everyday.) On a given day, if I don’t, I feel really crappy.

So far I’ve not seen a similar behavior when it comes to Learning.

So what made me change my attitude towards exercising but not learning?

• Most important thing, with exercising, both the outcome and output is easy to measure/judge.
  • There are some very easy measurements that can quickly show you the difference between being in-shape or not.
  • Did I workout for 30 mins everyday and is my weight stable? (Output)
  • Do I feel energized and fresh the whole day? (Outcome) (I can easily judge this.)
  • But with learning, how do I measure the outcome? Its a lot more vague and much more long term.
• Few things changed, I was in a company of people (at work, back home, amongst friends), who were all very conscious about staying in shape. Being surrounded by right people, who felt much better after staying in shape, reinforced my thinking and encouraged me to take the first step. Unfortunately its hard to find similar environment when it comes to learning.
• I was able to set some simple targets (run for 30 mins every day) and measure if I met the targets or not. If I did not meet those targets, the results were quite obviously immediately. Unfortunately when it comes to Learning I’m not able to find a simple model like this, esp. results being immediately obvious.
• Personally for me, exercising, which involves mostly physical work, is relatively easy even when I’m tired or feeling lazy. But its hard to learn under such conditions because learning requires high cognitive processing.
• There are many options for exercising, but I was able to quickly decide on running everyday as its the simplest thing I could do anywhere and still get a good workout. But when in comes to learning, we have so many options and its intimidating to even get started.
• Exercising usually is fun. Its enjoyable. There is an immediate positive feedback cycle. With learning that’s not always the case. Many times its frustrating.

When it comes to learning, I’ve tried to set various targets like:

• Everyday for 30 mins, either read a book or blogs or twitter or practice solving some design problems.
  • Problem is, even after doing this, I’m not sure if I’m really benefiting or not. May be in the long run, but not now.
• Write a blog post every other day. Hoping that, to write a meaningful blog post, I would go read something at least to clarify my thinking.
  • Problem is, I continue to write crap without reading much and I don’t write that regularly either.

What have you tried to keep up with your learning?

P.S: I learn a lot at my work, but its primarily driven by the need at hand. Which is good. But the broader learning and staying up-to-date with things is equally important and that’s what I’m talking about.

The fundamental attribution error (AKA correspondence bias/attribution effect) describes the tendency to over-value dispositional or personality-based explanations for the observed behaviors of others while under-valuing situational explanations for those behaviors.

For example, if Alice saw Bob trip over a rock and fall, Alice might consider Bob to be clumsy or careless (dispositional). If Alice later tripped over the same rock herself, she would be more likely to blame the placement of the rock (situational).

(Shameless rip-off from Wikipedia)

The funny part is, this error only occurs when we observe other’s behavior. We rarely apply fundamental attribution error to ourselves.

All of us are victims to this behavior daily. As Management, Coaches or Consultants, we need to be extra careful. Its easy to judge a team/company based on our personal explanation for the observed behavior and ignore the situational explanation.

For example, if a team of developers don’t meet their estimates, we might conclude that the developers are inexperienced and have not spent enough time estimating. They need to spend more time estimating, to get better at it. But if we look at the situational explanations, the developers don’t really understand what needs to be built, the person requesting for the feature is not clear what they expect, there might be a huge variety of ways in which the problem could be solved and so on. If we switch roles and try to play the developer’s role, we might be able to understand the situational/contextual explanation for the observed behavior.

I also see many people attribute poor team/company performance to lack of “Agile process”. They bring in the trainers and the coaches, who train and coach the team with standard “Agile practices”. Months/Years later, the team (what ever is left) have got good with process, but the product does not pull its weight and eventually dies out.

Fundamental Attribution Error is another reason why we see such vast spread abuse of Metrics in every field.

So how do we deal with this?

And I hear the Lean Extremists scream, 5 Whys…5 Whys…

Based on my personal experience, 5 Whys can also suffers from the same problem of Fundamental Attribution Error.

Being aware of the fundamental attribution error and other concepts like actor–observer bias and other forms of bias can help.

And in some cases, trial and error (brute force) seems to be the only answer.

By now you must have heard/read numerous people explain the pitfalls of multi-tasking and why its evil.

• Jeff Atwood’s blog on “The Multi-Tasking Myth“
• John Naish’s article on “Is multi-tasking bad for your brain? Experts reveal the hidden perils of juggling too many jobs“
• Clarke Ching’s blog on “Multitasking MAKES YOU STUPID“
• Joel Spolsky’s blog on “Human Task Switches Considered Harmful“
• Gina Trapani’s article on “Work Smart: Stop Multitasking and Start Doing One Thing Really Well“

Some people offer some decent advice on how to avoid Multi-tasking:

• Don’t Multi-task When You Can Use Chunking by F. John Reh
• The Pomodoro Technique forces you to focus on 1 task for 25 mins & then take a short break of 5 mins after every 25 mins (and a long break after 4 Pomodoros).
• The Dangers of Multitasking And How To Stop by Laura Earnest
• How NOT to Multitask – Work Simpler and Saner by Leo Babauta
• How to Avoid Multi Tasking on WikiHow

While I agree with everyone here. There are things in my daily life that contradicts (to some extent) what they are claiming to be universally true for all humans. For Ex:

• I listen to the news on the radio every morning while I brush my teeth
• I have interesting, deep conversations with friends/family while driving on Indian roads (driving itself involves multiple tasks. Driving on Indian roads adds a whole new dimension to it)
• The Indian Gods had attainted a whole new level of multi-tasking 😉

What I’m trying to highlight here is not all multi-tasking is bad/inefficient. If I take the driving example, I can do other tasks while I’m driving. My efficiency starts to go down if I’m driving to new destinations. It further goes down if I’m driving in a different lane-system and a different car.

Its important to note that multi-tasking does not hurt you much if the activities you’re performing are routine activities (embedded in your long-term memory and is referred to as muscle memory.) Multi-tasking becoming significantly exhausting, error-prone and inefficient if the tasks you are performing requires conscious processing/thought (.i.e. uses your prefrontal cortex intensively.) Five brain functions, understanding, deciding, recalling, memorizing and inhibiting thoughts, make up majority of conscious thought. Activities like planning, problem solving, communication, etc use these functions heavily. Hence multi-tasking on such activities is a bad idea.

Harold Pashler come up with a phenomenon called Dual-Task Interference. Via various experiments he showed that performing 2 cognitive tasks at once can reduce the cognitive capacity drastically. But if an individual performed 2 tasks, out of which only one task required conscious processing, then the cognitive capacity did not see the same drop.

There are simple activities which illustrates that one cannot normally carry out two tasks completely independently when each of them requires a choice of response.  When we try to do so, substantial delays occur in one or both tasks.  This is true even when neither task is anything that would be described as mentally challenging. Much research in this area argues that one particular mental operation is almost invariably carried out sequentially in tasks like this: the planning of responses.   The same is true of certain types of decision operations and memory retrievals.  On the other hand, the brain seems capable of perceiving stimuli while it is choosing a response, and actually producing motor responses in one task can overlap with the choice of a response in another.

Again, not all tasks requiring a choice of responses are subjected to this sort of processing bottleneck. Tasks that involve extremely “natural” mappings between stimuli and responses appear not to be.  For example, repeating words aloud as you hear them is a task most people can carry out in parallel with other tasks (McLeod & Posner, 1984).  The same is true of moving your eye to look at a spot (Pashler, Carrier & Hoffman, 1993).

Are humans capable of only uni-tasking? Not at all. If one of the tasks does not involve a choice of responses (e.g., if it merely involves repetitive rhythmic tapping, or requires perceiving and identifying stimuli without the need to decide on responses), interference is often reduced or even absent (subsequent demos on this site will illustrate this point).  Laboratory experiments in which response times are analyzed in detail have lent considerable support to the idea of a “central bottleneck” in response planning and indicated that other operations are often processed in parallel between the two tasks (for recent reviews, see H. Pashler, The Psychology of Attention, 1998, MIT Press; P. Jolicoeur, Journal of Experimental Psychology: Human perception and Performance, 1999, 25, 596-616).

Summary: Multi-tasking activities which require conscious processing is exhausting, error-prone and inefficient, hence a bad idea.