April 11th, 2015
I’m surprised when people think Agile is perfect and if there are any shortcomings, its not the problem with Agile, instead, it is the person/team/org’s understanding or implementation issue. Some where along the way, the aspect that “We are uncovering better ways of developing software” was lost and agile became this static, rule-based prescriptive and dogmatic cargo-cult thing.
IMHO Agile has made a significant difference (some of it a a placebo effect as well) to the software industry however it has some serious limitations when you try to apply in beyond simple CRUD based applications:
- Agile works well in linear or organised complexity domains where the problem is fairly well understood (static) and we need to find/evolve the solution iteratively and incrementally. But in domains, where:
- the problem itself is unknown or constantly shifting,
- the problem has a dozen or so variables that interact non-linearly. For ex:
- in life sciences where we’re trying to understanding ageing/growth
- in anti-terrorism where we have to deal with a crisis situation
- when simulating chaotic systems like Indian traffic system
- trying to predict outcomes in systems with distributed intelligence
applying agile values, principles and practices is not the best approach in these cases. We often find ourselves lacking the right kind of thought process and tools to be able to manage such project.
- Event though the Agile luminaries claim that Agile treats software development as a Complex Adaptive System, they actually try to apply techniques that work in a Complicated Domain.
- For example, given a problem, we analyse the problem, figure our a best-bet solution (set of practices), apply the solution, see what happens, do a retrospective and tweak the solution (inspect and adapt). This is how you work in a complicated domain. In a complex adaptive domain, we try a few independent safe-fail experiments to solve the problem, but most importantly we do all those experiments in parallel (set-based development approach), so we can really amplify good patterns and dampen bad patterns. We manage the emergence of beneficial patterns with attractors within boundaries. Its like running 5 parallel A/B tests and then coming up with a solution.
- Agile folks seems to claim that distributed development is hard and you should always prefer collocation. But what about thousands of successful open source projects built by people who’ve never met each other? We seem to be missing something here. Open source project model seems to be way better at motivating people by giving them autonomy, master and sense of purpose. Most agile projects are not able to match this.
- Today velocity and bunch of other vanity metric is killing agility. There seems to be so much focus on output and very little focus on outcome and learning. Agile has very little to offer in the space of customer development, business model validation, User experience and other important aspects required for a successful product launch. Which is what Lean-Startup movement is trying to address. This is clearly a limitation of Agile methods.
What’s your take?
January 26th, 2015
TL;DR: Definition of Done (DoD) is a checklist-driven project management practice which drives compliance and contract negotiation rather than collaboration and ownership. Its very easy for teams to go down rat-holes and start to gold-plate crap in the name of DoD. It encourages a downstream, service’s thinking mindset rather than a product engineering mindset (very output centric, rather than outcome/impact focused.) Also smells of lack of maturity and trust on the team. Bottom line: Its a wrong tool in the wrong people’s hand.
The Scrum Guide™ describes DoD as a tool for bringing transparency to the work a Scrum Team is performing. It is related more to the quality of a product, rather than its functionality. The DoD is usually a clear and concise list of requirements that a software Increment must adhere to for the team to call it complete.
They recommend that having a clear DoD helps Scrum Teams to:
- Work together more collaboratively, increases transparency, and ultimately results in the development of consistently higher quality software.
- Clarifies the responsibilities of story authors and implementors.
- Enables the Development Team to know how much work to select for a given Sprint.
- Encourages people to be clear about the scope of work.
- Enable transparency within the Scrum Team and helps to baseline progress on work items
- Helps visualize done on posters and/or electronic tools.
- Aids in tracking how many stories are done or unfinished.
- Expose work items that need attention
- Determine when an Increment is ready for release
Also according to them, DoD is not changed during a Sprint, but should change periodically between Sprints to reflect improvements the Development Team has made in its processes and capabilities to deliver software.
According to the LeSS website- DoD is an agreed list of criteria that the software will meet for each Product Backlog Item. Achieving this level of completeness requires the Team to perform a list of tasks. When all tasks are completed, the item is done. Don’t confuse DoD with acceptance criteria, which are specific conditions an individual item has to fulfil to be accepted. DoD applies uniformly to all Product Backlog items.
If you search online, you’ll find sample DoD for user stories more or less like this:
- Short Spec created
- Implemented/Unit Tests created
- Acceptance Tests created
- Code completed
- Unit tests run
- Code peer-reviewed or paired
- Code checked in
- Documentation updated
- 100% Acceptance tests passed
- Product Owner demo passed
- Known bugs fixed
- Upgrade verified while keeping all user data intact.
- Potentially releasable build available for download
- Summary of changes updated to include newly implemented features
- Inactive/unimplemented features hidden or greyed out (not executable)
- Unit tests written and green
- Source code committed on server
- Jenkins built version and all tests green
- Code review completed (or pair-programmed)
- How to Demo verified before presentation to Product Owner
- Ok from Product Owner
Do you see the problem with DoD? If not, read on:
- Checklist Driven: It feels like a hangover from checklist driven project management practices. It treats team members as dumb, checklist bots. Rather than treating them as smart individuals, who can work collaboratively to achieve a common goal.
- Compliance OVER Ownership: It drives compliance rather than ownership and entrepreneurship (making smart, informed, contextual decisions.)
- Wrong Focus: If you keep it simple, it sounds too basic or even lame to be written down. If you really focus on it, it feels very heavy handed and soaked in progress-talk. It seems like the problem DoD is trying to solve is lack of maturity and/or trust within a team. And if that’s your problem, then DoD is the wrong focus. For example, certain teams are not able to take end-to-end ownership of a feature. So instead of putting check-points (in the name of DoD) at each team’s level and being happy about some work being accomplished by each team, we should break down the barriers and enable the team to take end-to-end responsibility.
- Contract Negotiation OVER Collaboration: We believe in collaboration over contract negotiation. However DoD feels more like a contract. Teams waste a lot of time arguing on what is a good DoD. You’ll often find teams gold plating crap and then debating with the PO about why the story should be accepted. (Thanks to Alistar Cockburn for highlighting this point.)
- Output Centric: DoD is very output centric thought process, instead of focusing on the end-to-end value delivery (outcome/impact of what the team is working on.) It creates an illusion of “good progress”, while you could be driving off a cliff. It mismanages risks by delaying real validation from end users. We seem to focus more on Software creators (product owners, developers, etc.) rather than software users. Emphasis is more on improving the process (e.g. increasing story throughput) rather than improving the product. Ex: It helps with tracking done work rather than discovering and validating user’s needs. DoD is more concerned about “doing” rather than “learning”. (Thanks to Joshua Kerievsky for highlighting this point.)
- Lacks Product Engineering Mindset: Encourages more of a downstream Service’s thinking rather than a product engineering mindset. Unlike in the services business, in product engineering you are never done and the cycle does not stop at promoting code to high environment (staging environment). Studying whether the feature you just deployed has a real impact on the user community is more important than checking off a task from your sprint backlog.
What should we do instead?
Just get rid of DoD. Get the teams to collaborate with the Product Management team (and user community, if possible) to really understand the real needs and what is the least the team needs to do to solve the problem. I’ve coached several teams this way and we’ve really seen the team come up with creative ways to meet user’s need and take the ownership of end-to-end value delivery instead of gold-plating crap.
October 29th, 2014
Self-organised, self-managed and self-directed…do they mean the same thing or are they actually different concepts, where one might be more desirable over the other?
In the context of an “agile” team, people seemed to use these terms interchangeably. However, it’s important to note that there are subtle, yet worthwhile distinction between each.
A group of people working together in their own ways, toward a common goal, which is defined outside the team.
For example – the CEO of a company decides to launch a new product to address the needs of a specific target market. An initial team is assembled with a budget and high-level timelines. This team decides how they wanted to operate within the given budget. Team will do their own work scheduling, training, rewards and recognition, etc. They typically do a 360 review and rated other team members for salary appraisal. Also the team manages itself and its stake holders. They collectively play the manger’s role.
A group of people working together in their own ways, toward a common goal, which the team defines.
Usually, the team comes together for a common cause. In addition to the characteristics highlighted under the self-managed teams, a self-directed team also handles the actual compensation, discipline, and acts as a profit centre by defining its own future. In some sense, you can think of open-source projects resembling these characteristics. There is a big element of self-selection and built-in synergy.
Self-managed and self-directed have a noticeable differences in terms of autonomy and how they actually operate because of it. Listed below are attributes to consider when deciding how to structure your teams in your organisation:
||Receives goals from leadership and determines how to accomplish their goals
||Determines own goals and formulates a strategy to accomplish them
||Requires frequent open-communication from leadership on company goals and objectives to build employee commitment and increases morale
||Team itself creates an environment of high innovation, commitment, and motivation in team members
||Conducting effective meetings, problem solving, project planning, and team skills
||Decision making, entrepreneurship, resolving conflicts, and problem solving techniques
||Requires little supervision to track team’s progress and direction
||Prefers to work without supervision
||Can increase customer satisfaction through better response time in getting work done and resolving important customer problems
||Can delight customers by focusing on innovation, problem solving and reduced cycle time (local, informed decision making)
|Time to get team up & running
||Is relatively faster to get the teams to start working together, if the goal is given to them. Once they get started, they might face challenges due to lack of focus & motivation, but at least they will get started quickly
||Forming teams of high-caliber people, who can quickly converge on a common goal is hard. It can be expensive and time consuming to keep the team together and resolve conflicts. But once the team gels and get started, their performance is unmatchable.
||Requires some help from supporting teams like Learning and Development, Human Resource, etc.
||Pretty much self-contained; can manage with very little external support.
|Executive Leadership Involvement
||Requires them to guide, motivate and track team’s direction.
||Requires a system that provides two-way communication of corporate strategy between leaders and their teams.
Hopefully, this highlights the difference between self-managed and self-directed. What about self-organised?
First let’s understand what self-organisation, as a phenomenon means.
Self-organisation is a process where some form of global order or coordination arises out of the local interactions between the components of an initially disordered system. This process is spontaneous: it is not directed or controlled by any agent or subsystem inside or outside of the system; however, the laws followed by the process and its initial conditions may have been chosen or caused by an agent. It is often triggered by random fluctuations that are amplified by positive/negative feedback. The resulting organisation is wholly decentralised or distributed over all the components of the system. As such it is typically very robust and able to survive and self-repair substantial damage or perturbations.
Self-organisation occurs in a variety of physical, chemical, biological, social and cognitive systems. Common examples are crystallisation, the emergence of convection patterns in a liquid heated from below, chemical oscillators, the invisible hand of the market, swarming in groups of animals, and the way neural networks learn to recognise complex patterns. Self-organisation is also relevant in chemistry, where it has often been taken as being synonymous with self-assembly.
Sometimes the notion of self-organisation is conflated with that of the related concept of emergence. Properly defined, however, there may be instances of self-organization without emergence and emergence without self-organization, and it is clear from the literature that the phenomena are not the same. The link between emergence and self-organisation remains an active research question.
Self-organisation usually relies on three basic ingredients:
- Strong dynamical non-linearity, often though not necessarily involving positive and negative feedback
- Balance of exploitation and exploration
- Multiple interactions
Self-organisation in biology
Birds flocking, an example of self-organisation in biology. According to Scott Camazine – “In biological systems self-organisation is a process in which pattern at the global level of a system emerges solely from numerous interactions among the lower-level components of the system. Moreover, the rules specifying interactions among the system’s components are executed using only local information, without reference to the global pattern.”
Now let’s look at what is a self-organised team? Actually, the real question to ask is, what aspects of the team do they self-organise?
IMHO both self-managed and self-directed teams use self-organisation to achieve their objectives. Self-managed teams mostly self-organises to achieve their tasks, while self-directed team also uses self-organisation to form the team itself. It almost feels like self-managed/self-directed is one dimension (abstraction), while self-organised is a slightly different dimension (implementation.) While it feels like you cannot be self-managed or self-directed without self-organisation, I’m not 100% sure.
October 12th, 2014
Over the recent conferences, I’ve had several people ask me the follow:
I would like to better understand the expectations from the organising committee on the talk proposals. In particular, I would like your feedback on my talk submission so that I can work on improving the same.
I think, this is a very valid question:
What is the selection criteria for the talks?
I’ve been organising conferences for a decade now and following is my perspective:
In terms of the overarching themes or values, we look at the following during selection:
- Diversity – As a conference, we want to be more inclusive (different approaches, different programming languages, gender, countries, back-ground etc.)
- Balance – We want to strike a good balance between different types of presentations (expert talks, experience reports, tutorials, workshops, etc.) and different types of experience the speakers bring to the conference.
- Equality – We encourage more students and women speakers. We won’t select a stupid proposal just because it came from a student or a female speaker. But given we have to pick 1 out of 2 equal proposal, we’ll pick the one, which was proposed by a student or a female speaker.
- Practicality - People come to a conference to learn, network, have an experience and leave motivated. Proposals which directly help this are always preferred. While a little bit of theory is good, but if the proposal lacks practical application, it does not really help the participants. Also people learn more by doing rather than listening. If proposals has an element of “learn by doing” it wins over other proposals. Take people on a learning journey.
- Opportunity – While we want to ensure the conference has at least 2/3 rock solid speakers, we also want to give an opportunity to new speakers, who have real potential
- Originality - Original ideas wins hands-on from copied one. People always prefer listing to an idea from its creator rather than second or third person. However, you might have taken an idea and tweaked it in your context. You would have gained an insight by doing so. And certainly all of us want to hear your first-hand experience, even though you were not the creator of the original idea. We are looking for Thought-Leadership.
- Radical Ideas – We really respect people, who want to push the boundaries and challenge the status quo. We have a soft-corner for unconventional ideas and will try our best to support them and bring awareness to their work.
- Demand – Votes on a proposal and buzz on social media gives us an idea of how many people are really interested in the topic. (We fully understand votes can be gamed, but we’ve a system that can eliminate some bogus votes and use different types of patterns to give us a decent sense of the real demand.)
Once the proposal fits into our value system, here are some basic/obvious stuff we expect when we look at the proposal in the submission system:
- Is the Title matching the Abstract?
- Under the Outline/Structure of the Session, will the time break-up for each sub-topic will do justice to the topic?
- Is there a logical sequencing/progression of the topics?
- Has the speaker selected the right session type and duration for the topic? For Ex: 60 mins talk might be very boring.
- Has the speaker selected the best matching Theme/Topic/Category for the proposal?
- Is the Target Audience specific and correct? Also does it match with the Session Level?
- Is the Learning Outcome clearly articulated? Ideally 3-5 points, one of each line.
- Based on the Outline/Structure, will the speaker be able to achieve the Learning Outcomes?
- Based on the presentation link, does the speaker have good quality content and good way to present it?
- Based on the video link, does the speaker have a good presentation (edutainment) skills? Will the speaker be able to hold the attention of a large audience?
- Based on the additional links, does the speaker have subject matter expertise and thought leadership on the proposed topic?
- Are the Labels/Tags meaningful?
Proposal stands the best chance to be selected, if it’s unique, fully flushed, ready-to-go. Speaker please ensure to provide links to your:
- previous conference or user group presentations
- open source project contributions
- slides & videos of (present/past) presentations (other conferences or local user group or in-office)
- blog posts or articles on this topic
- and so on…
When selecting a proposal, we pay attention not only to the quality of the proposal, but also quality of the speaker, .i.e. whether the speaker will be able to effectively present/share their knowledge with others. Hence past speaking experience (videos & slides) are extremely important. If you don’t have a video from past conference presentation, that’s fine. Try to setup google hangout in one of your upcoming local user group meeting or internal office meeting, where you are presenting and share that link. This will give the committee a feel for your presentation skills and subject matter expertise.
While this might look very demanding, it is extremely important to ensure we put together a program which is top-notch.
October 3rd, 2014
Many teams suffer daily due to slow CI builds. The teams certainly realise the pain, but don’t necessarily take any corrective action. The most common excuse is we don’t have time or we don’t think it can get better than this.
Following are some key principles, I’ve used when confronted with long running builds:
- Focus on the Bottlenecks – Profile your builds to find the real culprits. Fixing them will help the most. IMHE I’ve seen the 80-20 rule apply here. Fixing 20% of the bottlenecks will give you 80% gain in speed.
- Divide and Conquer – Turn large monolithic builds into smaller, more focused builds. This would typically lead to restructuring your project into smaller modules or projects, which is a good version control practice anyway. Also most CI servers support a build pipeline, which will help you hookup all these smaller builds together.
- Turn Sequential Tasks to Parallel Tasks – By breaking your builds into smaller builds, you can now run them in parallel. You can also distribute the tasks across multiple slave machines. Also consider running your tests in parallel. Many static analysis tools can run in parallel.
- Reuse – Don’t create/start from scratch if you can avoid it. For ex: have pre-compiled code (jars) for dependent code instead of building it every time, esp. if it rarely changes. Set up your target env as a VM and keep it ready. Use a database dump for your seed data, instead of building it from an empty DB every time. Many times we use incremental compile/build, instead of clean builds.
- Avoid/Minimise IO (Disk & Network) – IOs can be a huge bottleneck. Turn down logging when running your builds. Preference using an in-process & in-memory DB, consider tmpfs for in-memory file system.
- Fail Fast – We want our builds to give us fast feedback. Hence its very important to prioritise your build tasks based on what is most likely to fail first. In fact long back we had started a project called ProTest, which helps you prioritise your tests on which test is most likely to fail.
- Push unnecessary stuff to a separate build – Things like JavaDocs can be done nightly
- Once and Only Once – avoid unnecessary duplication in steps. For ex: copying src files or jars to another location, creating a new Jenkins workspace every build, empty DB creation, etc.
- Reduce Noise – remove unnecessary data and file. Work on a minimal, yet apt set. Turn down logging levels.
- Time is Money -I guess I’m stating the obvious. But using newer, faster tools is actually cheaper. Moving from CVS/SVN to Git can speed up your build, newer testing frameworks are faster. Also Hardware is getting cheaper day by day, while developer’s cost is going up day by day. Invest in good hardware like SSD, Faster Multi-core CPUs, better RAM, etc. It would be way cheaper than your team waiting for the builds.
- Profile, Understand and Configure – Ignorance can be fatal. When it comes to build, you must profile your build to find the bottleneck. Go deeper to understand what is going on. And then based on data, configure your environment. For ex: setting the right OS parameters, set the right compiler flags can make a noticeable difference.
- Keep an Open Mind – Many times, you will find the real culprits might be some totally unrelated part of your environment. Many times we also find poorly written code which can slow things down. One needs to keep an open mind.
Are there any other principles you’ve used?
BTW Ashish and I plan to present this topic at the upcoming Agile Pune 2014 Conference. Would love to see you there.
July 28th, 2014
A quick update on upcoming conferences:
- Selenium Conf 2014 – 4th Annual Selenium Conference. Draft program schedule is now available at http://seleniumconf.org/#program. Also you’ll notice that the registration for the 4-pre-conference workshops are also open now. We’ve limited seats, grab them now at http://booking.agilefaqs.com/selenium-conf-2014
- Functional Conf 2014 – 1st Functional Programming Conference in India. Draft program schedule is now available at http://functionalconf.com/#program. Last few smart registration seats are left. Grab them at http://booking.agilefaqs.com/functional-conf-2014
- Agile Pune 2014 – July 31st is the last day to submit your proposal for the conference. We have already received 27 proposal. You can view them at http://confengine.com/agile-pune-2014. Also last few seats are left for the Early-Bird registration. Book your seats at http://booking.agilefaqs.com/agile-pune-2014
- Agile DC 2014 is accepting proposal till Aug 15th. So far, we have 33 great proposals submitted. Check them out at http://confengine.com/agiledc
Blast from the past: All the video from Agile India 2014 Conference are now publicly available here: http://confengine.com/agile-india-2014/schedule.
June 24th, 2014
We are delighted to present Linda Rising and Joshua Kerievsky, as our keynote speakers for the upcoming Agile Pune 2014 Conference. The conference will be hosted at Hyatt Regency, Pune on Nov 21st and 22nd.
The Agile Pune 2014 is a volunteer-run, non-profit event organised by the Agile Software Community of India (ASCI). The goal of the conference is to bring together Agile enthusiasts from around the world to share ideas, socialise, and work together on advancing the state of Agile/Lean Software development.
Simplicity, quick feedback cycles, systems thinking, mistake proofing, transformational leadership, building quality in, relentless improvement via inspect and adapt, evolutionary design, cross-functional collaboration, sustainable pace, self-organisation and safe-fail experimentation are some of the core tenants of the Agile and Lean mindset. They help teams embrace uncertainty and make them more change resilient. This conference is dedicated to deep-dive on these topics.
Agile Pune 2014 is a two-day conference, starting on Nov 21st (Friday), where experts and practitioners from around the world will share their experience on topics related to our theme, “Action Precedes Clarity”. The conference will host 3 parallel tracks. We also plan to host pre-conference workshops to help you get world-class training directly from our international experts at affordable rates.
If you are interested in presenting at the Agile Pune 2014 conf, please submit your proposals at http://confengine.com/agile-pune-2014
Super early bird registrations starts today. Register now at http://booking.agilefaqs.com/agile-pune-2014
To know more about the conference, please visit http://pune.agileindia.org
June 19th, 2014
Remember the dot-com days of Webvan and Pets.com? We took traditional businesses and gave then an online presence. Rapidly acquiring a large customer base was the sole goal of many dot-coms. “If we can get enough users, we can easily figure out how to monetize it.” And all of this made perfect sense expressed in dollars and cents. I know people who melted down Yahoo Finance’s servers by checking for their favourite stocks prices throughout the day, calculating their (paper) net worth in real time. If you were not part of this madness, you were certainly considered stupid.
But then on March 10, 2000, the perspective changed. Suddenly it became clear that this was really a bubble. Without having real profits (or even revenue/cash-flow), it was really just a house of cards. In hindsight, the entire dot-com burst made perfect sense. But why wasn’t this obvious to everyone (including me) to start with?
In complex adaptive system, the causality is retrospectively coherent. .i.e. hindsight does not lead to foresight. When we look back at the events, we can (relatively) easily construct a theory to explain the rationale behind the occurrence of these events. In fact, when we look back, the reasons are so obvious that one can easily be fooled into believing that “Only if we spend more time, carefully analysing and thinking through the situation at hand, we can completely avoid unwanted events in future.” Yet, time and again, we’ve always been caught by surprise and it almost appears to be impossible to predict such events ahead of time. Call it the Black Swan effect or whatever name you fancy.
This effect gives rise to a classic management dilemma – Predictability Paradox(pdf). In the zeal to improve the effectiveness and reliability of software development, managers institutionalise practices that unfortunately decrease, rather than increase, the predictability of the product’s success. Most companies spend an awful lot of effort and money to analyse the past, derive patterns and best practices, set targets and create processes to prevent past failure and produce ideal future goals. If software development was highly structured, if we had a stable environment and we had a good data points from million other projects, this approach might work. But for software development, which is a creative-problem solving domain, with high levels of uncertainty and each project having an unique context, these techniques (best practices) are rather dangerous.
In our domain,
- We need to break the vague problem down into small safe-fail experiments.
- Then execute each experiment in short iterative and incremental cycles.
- We need to focus on tight feedback loops, which will help us adapt & co-evolve the system. (We cannot be stuck with analysis paralysis.)
- We need to probe the system with experiments and find evolutionary practices.
- And then apply these practices in a given context, for a short duration.
- Speed and Sustainability are extremely important factors.
This is what I mean when I say “Action Precedes Clarity”.