Software Development is a complex process and I don’t mean to use the word “Complex” in a classic dictionary way but in accordance with Dave Snowden’s Cynefin framework. In my experience of the failed the software projects, most of them made the mistake of treating software development as Obvious (Best Practice) or Complicated (Good Practice). It is not, software development is mostly Complex or Obvious that quickly shifts to Chaotic. As always we can always claim to be wise in retrospect but real wisdom lies in the fact that we learn from the mistakes of the past and make improvements.

Image by Snowded - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=33783436
But it is easier said than done, I have seen even the sharpest of the minds falling to catch this that eventually led to project failures. Why does it happen, why smart managers and teams not able to handle it? Let me put it in most succinct way possible and then elaborate on that.
“They fail because they fail to distinguish between Business Process and Technical Process, and the relationship between them.”
Here is a 2 minutes summary:
For example, if you are developing a software for Loan repayment processing module, you can investigate and collaborate with business to know the business process flow including the players, events and state transitions involved. Now you have all “known knowns” in place and you are also well placed to handle the any changes to process as they come along. This will be placed in the Obvious domain of Cynefin framework.
However if you were to develop software for the process above then we have to think about the software development process and all the technical aspects that go with it. We always have some best practices to start with, basic design and architecture is prepared and development starts with some “knowns unknown” which we typically manage as risks. This would place software development in the Complicated domain. But when it comes to technology there are always “unknown unknowns” that we need to acknowledge and be ready to face them as they surface. If we are not doing that then soon we will have software development that will drift quickly from Complicated to Complex to Chaotic while the disorders rule. And that’s what happens when the software projects for a seemingly simple business process fails where we fall over the edge to Chaotic domain.
The Agile Process and Framework setup is all about Good and Best Practices to follow, however Technical Agility involves the emergent practices that we get through experimentation. Integrating Technical Agility as part of the Agile process setup gives the means to make it happen and not do it as an afterthought.
Technical Agility gives the teams power to thrive in uncertainty and not just survive.
Here is longer detailed elaboration to it:
While Lean and Agile methods are quite useful, working with any Agile framework as suitable to your needs is a commendable way forward. But we can not stop at that, if anything this is just an start to our Agile journey.
The same Lean-Agile principles and patterns that enable the fast flow of work in the business processes are equally applicable to technology. We need to define our technology value stream as the process required to convert a business hypothesis into a technology-enabled service that delivers value to the customer.
The formulation of a business objective or the hypothesis are the inception of the Software Development process and we accept the work in Development, adding it to our committed backlog of work. Our goal being to develop Software at lower cost, with fewer defects, with higher productivity, and with much higher return on investment. In order to achieve this we need all the stakeholders to work together and collaborate. It’s a beautiful synchronisation, almost like a dance form or a symphony with Agile process, Business Process and Technical Process coming together and creating a rhythm. I call this rhythm as being Agile and Technical Agility is an indispensable part of it.
Technical Agility starts much before a single line of code is written and continues much after the coding has been completed. A big part of Technical Agility is about giving up old, ineffective technical habits in favour of new ones that work.
There are best practices and good practices that we can make use of, the list if pretty big and ever growing, such as BDD, TDD, Pair Programming, Version Control, Trunk based development, Code Analysis, Coding Standards, Continuous Integration, Continuous Deployment, Automated Testing, Testing Coverage, Release on Demand, Dark Launches, Immutable Infrastructure, Full-stack telemetry.. the list is indeed endless. Ultimately our goal is to produce bug free software product in stipulated time that adds value to the client. However, in context of the Cynefin framework, along with the good and best practices (Obvious and Complicated domains) there will always be Complex parts that would call for creating safe to fail experiments <link to Fail-fast blog>, where we try different approaches, evaluate the results, and then iterate through the inspect and adapt cycles of the experiments along the way to see what works best in your context.
Agile is iterative so is the testing. In any Agile model the requirement gathering and documentation, and design activities, and development activities, are iterative in nature. So must testing activities. If Testing team is not adapting to this then it will lead to failure.
Testing activities need to start along with the requirement elicitation unless we want to treat Testing as phase and suffer from Mini-Waterfall guised as Agile. As these requirements are ever evolving, there will not be a time where comprehensive requirements specification will be made available, therefore testing strategies can not wait for “frozen” specifications. We need to work as a whole team during the requirements gathering and grooming activities and create test strategies on the go. With Technical Agility we can make our Test Automation robust to reduce the regression testing time, make the whole process more repeatable, reliable and configurable (Testing Frameworks). This will lead to our Continuous Testing strategy without which Continuous Integration and Continuous Deployment/Delivery is not possible.
When Business Process is under Complex or Complicated (or Chaotic) domain, we need to keep Agile Process and Technical Agility in Obvious Domain. By that I mean to say that we can not experiment at the both Business and Agility end, we need to stabilize one to support the other.
All Agile frameworks are inherently emergent, they aims at solving complex problems by using the Probe, Sense, Respond method. It would not be accurate to say that all software projects are complex, I would say that most projects that are building something new that is non-trivial in nature, would have many uncertainties that can only be explained in retrospect.
The software projects can go through the cycle of simple, complicated and complex problems and Agile comes handy in the discovery process as it is based upon inspect and adapt. As we are incremental and iterative with small batches, the cost of change is smaller and very capable of handling complex issues, this does come at the cost of staff utilisation (or efficiency) especially during the complex phase.
The software development process over the years has become standardized yet there are “unknowns unknowns” which make us revisit and continuously evolve our plans. Today we can gather sufficient data using continuous monitoring and full-stack telemetry. We can sense and analyse and to some extent even predict the outcome that can help us adapt our plans. This also gives us tools to help our marketing and user experience, A/B Testing, Blue-Green Deployment and Canary releases are prominently used by enhancing the technical agility. Not so long ago, the marketing teams were could not do all this because it required costly engineering and IT resources. Today, technical agility enables this without depending on engineers and give powerful tools for making data-driven decisions. More power to Technical Agility is more power to business.
